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properly estimate saople design-based covariance matrices is; not 
generally available to NAEP data users. Design effect methods for 
adjusting test statistics obtained from standard statistical methods 
which implicitly assume simple random sampling from an infinite 
population are presented with several new decompositions obtained 
which display the effects of multistage clustering, stratification, 
and unequal weighting on the covariance matrix. A comparison of 
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Abstract 

Design Effects and the Analysis of Survey Data 

by 

Ra5,ph E. Folsom 
and 

Rick L. Williams 



The National Assessment of Educational Progress (NAEP) , like most 
large national surveys, employs a complex stratified multistage unequal 
probability sample. When properly accounted for in the analysis, the NAEP 
sample design provides a rigorous justification for extending survey 
results to the entire U.S. student target population. This paper reviews 
recent developments in the analysis of data from complex surveys which 
provide a straightforward method for taking account of the sample design 
through proper estimation of subpopulation estimates and their covariance 
matrix. Relationships among subpopulations can then be evaluated via large 
^sample Wald statistics assumed to be asymptotically distributed as central 
chi-squared random variables. 

While thiese methods provide a mechanism for analysing NAEP data, the 
computer software required to properly estimate sample design-based cp- 
variahce matrices is not generally available to NAEP data users. Recent 
literature has suggested methods for adjusting test statistics obtained 
from standard statistical methods which implicitly" assume simple random 
sampling from an infinite population. These so called design effect 
adjustments are reviewed and. several new decompositions obtained which 
display the effects^ of multistage clustering, stratification and unequal 
weighting on the covariance matrix. 

Finally, an empirical comparison is presented of asymptotically valid 
sample design-based chi-squared tests versus analogous simple random samp- 
ling tests and design effect adjusted tests. These comparisons are made 
for linear contrasts of domain means and proportions as ytell as for linear 
models fitted to the domain estimates via weighted least squares. The data 
were taken from the NAEP 1977-78 Mathematics assessment for 9-, 13- and 
17-year-olds. For these data, the analyses indicate that the design effects 
type adjustments of standard test statistics are not stable and are gener- 
ally too conservative to be of practical value. 
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• 1. INTRODUCTION 

The .National Assessment data base provides a wealth of information on 
the way student ability to correctly^ answer ^NAEP test items relates to- 
student background and school ^environment variables. When properly taken 
account of in the analysis, the Gorhplex probability sample design used to 

0 

collect NAEP data provides a rigorous justification for extending survey 
results to the entire U.S. .stud'ent population. Recent developments in the 
analysis of categorical data from complex surveys provide a straight for- 
ward methodology for taking account of sample design through the proper 
estimation of subpopulation proportions (domain P-values) and their covari- 
ance matrices [Koch, Freeman,' and Freeman (1975) ] . These vectors of sub- 
population P-values are then fit to linear models in the domain defining 
variables using the sample design based covariance matrix to calculate 

weighted least squares fits. Wald statistics that take the form of 

\ 

2 

Hotelling's multivariate T statistic are then used to test for the good- 
ness of fit of the model and to subsequently^ test for the significance of 
model effects. The following chapter surveying theoretical results begins 
with a section on the ' Wald statistic/weighted least squares theory for 
testing hypotheses about NAEP domain p-values. 

While the weighted least squares methodology provides a straight 
forward solution to the NAEP p-value analysis problem, the computer soft- 
ware required to properly estimate sample design based covariance matrices 
is not generally available to . NAEP data users. Section 2.2 summarizes 
methods explored by Fellegi (1980) and Rao and Scott (1981) for adjusting 
test statistics derived from standard statistical software packages. The 



0 

) ^ 

standard statistical software packages employ either implicitly or expli- 
citly covariance matrix calculations appropriate for simple random samples. 
The adjustments proposed by Fellegi, Rao and Scott involve dividing the 
standard chi-squared statistic by a generalized design effect CDeff) summa- 
rizing the ratio of sample design based P-value variances and covariances 
divided by their respective simple' random sampling variances and covari- 
ances. To display the sample design effects of multi-stage clustering, 
stratification, and unequal weighting on the generalized Deff, a design 
effect model identity is developed in Section 2.3 for the P-value co- 
variance matrix and for linear contrasts among sample P-values. The proper 
sample 'design based inference for NAEP balanced fits obtained as dummy 
variable regression coefficients is presented in Section 2.4 along with the 
analogous generalized Deff adjustment. Since much of the ..descriptive 
analysis of NAEP data utilizes subpopulation averages across several indi- 
vidual item P-values, the proper covariance matrix estimation and gene- 
ralized Deff adjustment ^methods for such statistics are explored in Sec- 
tion 2.5. 

\ 
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2. THEORETICAL RESULTS 

2.1 The Wald Statistic/Weighted Least-Square Theory 

To illustrate the weighted least-squares approach to the analysis of 
NAEP domain P-values, Consider a'^^single questionnaire ^^item with response 
categories labelled r = 1,2,...,R. Suppose further that there are D sub- 
population domains of- interest labelled d = 1,2,...,D. These domains can 
be viewed as student subpopulations formed by a cross-classification of 
background variables including Race, Sex, Region, Type of Community, and 
Parents Education. Let X^(t) depict a subpopulation indicator variable 
taking the value 1 when student t belongs to domain d and zero otherwise. 
Similarly, let Y^(t) denote response indicator variable taking the value 
1 when student t gives response r to the specified item and zero otherwise. 
With the U.S. student population size for a particular age class denoted by 
M, the population count .of domain d members giving response r is 



M 

Y(dr) = 1 X^(t) Y^(t) . 



The U.S. student population size for subpopualtion d is specified as ^ . 

M 

X(d) = 1 X^(t) . 

The proportion of domain d members giving response r is then defined as the 
ratio 

, P(dr)- = Y(dr)/X(d) . 
The NAEP sample estimates for these student subpopulation response propor- 
tions (domain P-values) have the form 
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n 

m 



P(dr) = [ 1 W(k) X,(k) Y fk)] - [ 1 W(k) X,(k)] 
k=l k=l 

where m denotes the number of sample students and W(k) is a sample, weight*^ 
incorporating the reciprocal of the sample student k inclusion probability 
7t(k) and various adjustments for sample school and student nonresporise . 

To simplify the notation, consider the column vector P(d) of (R-l) 
subpopulation d response proportions consisting of the first (R-1) esti- 
mates defined above. Stacking these domain specific vectors on top of one 
another, a single column vector P with D(R-l) elements is produced. Let 
^Vp(DES) denote the D(R-l) by D(R-l) estimated covariance matrix derived for 
P according to one of the three asympotically equivalent methods of vari- 
ance estimation for nonlinear statistics from complex probability samples, 
namely the Taylor Series linearization (TbL) method, balanced repeated 
replication (BRR) , and jackknife replication (JKR) . KreWski'and Rao (i978) 
have established limiting conditions for the asymptotic equivalence of 

o 

these covariance matrix est'imation methods. A^central limit Xheoi^ esta- 
blishing conditions for " the asymptotic nomality of studentized P-values 

t(dr) = [P(dr) - P(dr)] / [Vp^gCdr)]"^ 
is also presented by Krewski and Rao when any of the three linearization 
methods is used to approximate the sample design based variance (dr) 
.for the ratio statistic P(dr). Such a central limit theorem provides the 
theoretical justification fot assuming that the vector P of estimated 
domain P-values will be distributed approximately as a D(R-l)- variate 
normal vector with mean P and covariance matrix Vp(DES) . 

Assuming that the conditions for asymptotic normality apply, weighted 
least squares methods following Grizzle, Starmer, and Koch (1969) can be 
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used to fit linear model^sAto futictions of the estimated domain P-values. 
Let G(P) denote a vector of A continuous, linearly independent functions 

with partial derivatives through second order. Define the A by D(R-l) 

\ 

matrix of partial derivatives with row a denoted by 

H^(P) = aG^/ap - (ao^/aPi, • • • , ao^/aPp^j^.^)). 

The matrix of these partial derivatives is evaluated at the estimated P 
values to define 

H = H (P) . 
Consider, for example, the logit function where 

with a = 1,2,...D indexing the D domain P-values associated with the typical 
R=2 (correct-incorrect) item response breakdown. For this logistic function 

= (0,0,...,1/P^(1-P^), . . . 0) . 

Now, with G = G(P) denoting the sample estimate for the vector of A func- 
tions, one notes that G is asymptotically A-variate normal with mean vector 
G = G(P) and asymptotic covariance matrix o 

y^(DES) = [H Vp(DES) H^]. 
where denotes the transpose of H. A consistent estimate for V^(DES) is 

Vg(DES) = °[H Vp(DES) H^] . 

C 

One can now proceed to fit a general linear model of the forrn 

where the columns of X specify selected main effect and interaction con- 
trasts in terms of the domain defining Race, Sex, Type of Community and 



Parents Education variables. The' asymptotically efficient BAN (Best 
Asymptotically Normal) estimator for the coefficient vector g is then 
g = [X' V^(DES) X] X V^(DES) G 

S 

with asymptotic covariance matrix 

,\ Vp(DE&) = [X^ V^(DES)^^ X]^^ . 

To test the fit of the model, that is"" Hq: G(P) = X g, the residual quadra- 
td-G form o A ^ 

T^(Fit) = (G - X g)^ V^(DES)"^ (G - X g) 

is a Wald (1943) statistic which has the form of Hote.lling's multivariate' 

2 2 2 

T ..statistic. Asymptotically. T (Fit) is distributed as Chi-Square (x ) 

with degrees of freedom df = rank of X under the null hypothesis. For 

subasymptotic situations where the number of replicates used- to form the 

covariance matrix estimator does not substantially exceed the rank of X,' a 

2 

transformation of T to Snedecor's F may be appropriate. This leads to 

2 

. F(df , L - df + 1) = (L - df + 1) T (Fit)/df (L) 
where L is the titimber of degrees of freedom suggested by the .quadratic form 
used to estimate yp(DES) and df is the rank of X. This transformed Wald 
Statistic is compared against critical values of Sne decor ' s F with df 
numerator and L - df + 1 denominator degrees of freedom. 

Failing to reject Hq: ,G(P) = X g, one can entertain linear hypotheses 

0 

of the form Eq: £ g = $> with ^ denoting a null vector. The associated 
Wald Statistic is ' - 
T^q) = (C g)^ [C Vp(DES) c''^]"^ (C g) 

which, is asymptotically (j-ank of C) under the uull hypothesis. The F 
transformed alternative ^ 

F(c, L - c + 1) = (L - c + 1) t'(C)/cL 



ERIC 



is compared with critical values jof Senedecor's F with c = rank of C numera- 
tor degrees of freedom and (L - c + 1) denominator degrees of freedom. In 
the next section the approximate methods of Rao and Scott (1979) based on ^ 
simple random sampling covariance matrices Vp(SRS) and generalized design 
effect (deff) adjustments are explored. 
2.2 Generalized Design Effect Methods 

Rao and Scott (1981) considered the asymptotic distribution of Wald ° 
Statistics based on SRS covariance matrices for testing general hypotheses^- 
of the form 

Hq: G (£) = 0. 3 = 1,2,. . . .A 

where P can be viewed as the national response distribution for a specified 
NAEP exercise. This vector of universe level proportions corresp->ii is to 
the P(d) domain specific item response proportions introduced earlier with 
P(dr) denoting the domain or subpopulation proportion selecting the coded 
response option r = 1,2, . . . , (R-1) . Since the P(dr) sum to one over all R 
mutually exclusive response levels, only-(R-l) of the parameters is required 
'to fully characterize the response : distribution. While Rao and Scott's 
results focus on the single population problem, they can be extended simply 

c 

•> 

to the multiple subpopulation or domain problem by allowing P to represents 
the extended vector ^ 
P^ = < P^(l), . . ., P^(d), . . P^(D) > . 

Letting 

H (P) = 30 (p)/ap ■ . 

'^a a 

= [3G (P)/3P(11), . . .... 30 (P)/3P(D,R-1)] 

denote the vector of partial derivatives of G^(P) with respect to the 
D(R-l) elements of P, the population of response proportions, then 
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(G) = [H V (SRS) H^]"^ G 

SRS p 

is the SRS based Wald statistic for testing the hypothesis that the vector 
T 

of A functions G = [G^CP) , . . . , are simultaneously zero with H 

denoting the matrix of all partial derivatives H(P) evaluated at P = P. 
Interest in the SRS based statistic stems from the simple computational 
form for Vp(SRS). The simple random sampling covariance matrix for P is 
approximated as a block diagonal matrix with (R-1) by (R-1) blocks of the 
form 

V (SRS) = {diag [P(d)] - P(d) P^(d)}/m(d) 



where m(d) is th^^ domain d sample size and diag [P(d)] is the (R-1) by 
(R--1) diagonal matrix with diagonal elements P(dr). This SRS based covari- 
ance matrix is formed simply from the weighted domain P-values and the 
observed domain sample sizes. For the typical NAEP ^analysis of correct 
responses with R = 2, - 

V^(SRS) = P(d) [l-P(d)]/m(d) 

and Vp(SRS) is a D by D diagonal matrix with the V^(SRS) quantities on the 
diagonal. Under the null hypothesis Hq: G(P) = c|>, 

r . A , 

x^CG) 1 6 x\ 

^,SRS a=l a 

where tlCe 6^s are the eigenvalues of 

[H Vp(SRS) H^]""^ [H Vp(DES) H*^] , ^ 

6i > - . • ^^A^ Oy the x^ 's are independent Xi (single degree of freedom 
A a 

chi-squared) random variables and 6 is the value of- 6 under Ho. 

oa a - ^ 

Rao and Scott point out that the 6 's can be interpreted as design 

a 

effects of linear combinations L of the components of H P. Letting A 



denote the a-th eigenvalue of 

Vp(SRS) Vp(DES) 

then the 5 eigenvalues can be bounded by the \ eigenvalues as follows: 



for a = 1, A, since the are particular linear combinations of the 

P(dr)'s. Using a result in Anderson and Das Gupta' (1963), Rao and Scott 
establish .more precise bounds for the 6^ in terms of the"\^; namely, 

\-^a- ^D(R-l)-A+a ' 

* 2 

This inequality is useful for specifying an alternative to the following. X 

"*t:es"r^t'ari~s'tli:~irro"^^^^ ; ~7 

- \ [«oa/«d.] ^a 
a=l 

where 6^ denotes an estimate of the mean eigenvalue 6^ with^the 6^ depict- 
ing eigenvalues of 

[H Vp(SRS)^H^]"^ [H Vp(DES) H^] . 

.Since the estimation of the 6 and associated 6. require knowledge of the 

a ^ 

full design based covariance matrix , thei^.e is no real utility in usiiig this 

0 

approximation when one could just as well use the appropriate design based 
WaLd statistic. Using the sharp bounds for S , one notes that 6^ lies 

a a . . 

between the average of the A largest \^'s and the mean of the^ A smallest 

.\ 's. This implies that 6 should get close to \^ as thd number of G 
a , • 

functions (A) approaches D(R-l). With A = D(R-l) independent functions 

ft 

such that H is nonsingular it is clear that 6. = X.. If the'\^'s show 
little variation, so that the mean of the A largest and A smallest \'s are 
similar, then one can 'also expect that 6. = \. The advantage of using \. 
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instead of 5^ to adjust^ Chi-Square X^gj^g(G) is the ease of estimating . 
We note that 

Vp(SRS) = BLK-DIAG [{di^g [P.(d)] - P(d) P(d)l/m(d)] 

is block diagonal with blocks comprised of the domain specific multinomial 

covariance matrices Vp^^^(SRS)- Therefore, extending* results of Rao and 

Scott one .obtains 

\. = . trace {Vp(SRS)" Vp(DES)} / D(I^--1.). 

D . ^ 1 ^ 

trace Vp^^^ (SRS)" Vp^^^ (DES) / D(R-l) 

D R-1 

= II' V(dr|DES) m(d) / P(dr) D(R-l) 
d=l r=l 

D R-1 ^' - 
= t 1 [l-P(dr)] DEFF (dr) / D(R-l) 

d=l r=l u 

where 

- DEFF(dr) = (dr|DES) / {P(dr) [l-P(dr)] / m(d)} 
is the design effect for the cell proportion P(dr)< 

Returning to the NAEP correct-incoi rect response pattern (R=2), \^ 
simplifies to 

a ^ 

\ = I DEFF (dr) / D, 
d=l 

the mean of the domain specific design effects. This result follows from 
the diagonal form for 

Vp(SRS)"^ = diag [m(d)/{P(d) [l-P(d)]}] . 
In either case, the generalized design effect is a simple function of 
domain P-value design effects. When the design effects for subpopulation 
P-values, P{dr), are published then \/ can be formed without knowledge of 

. 16 
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the design based covariances between response porportions P(dr) and P(d'r') 
from different subpopulations (dj^d^). In the following chapter, numerical 
comparisons of the adjusted SRS based statistics 

xJjjjCG) =;x|j,s (G) /X. 

and the associated design based xJ^g^S^ ^^^^ statistics are explored. 

To model the effects of sample design features like stratification 
clustering and unequal weighting on the X^j^g^^-* statistic, one can develop 
a model for the generalized design effect matrix 

Vp(SRS)"^ Vp(DES) 

Consider, for example, a two-stage design with S primary frame units (PFU's), 
say schools, with M(s) secondary units (students) in the s-th PFU such 

S ' 

that 1 M(s) = M. A with replacement selection of n primary sampling units 
s=l 

(PSU's) is first made with single draw probabilities (t)(s) = [M(s)/M]. 

A subsequent with replacement simple random sample of m second stage units 

is then drawn from each sample PSU. For a single universal domain, Rao and 

Scott display the following partitioning for 

S »P 
Vp(DES) = Vp(SRS) + (m-1) 1 (|)(s) (P^ - P) (P^ - P) /nm 

<o S^l 

. = Vp(SRS) [I + (m-l)R] 
where (t>(s) = M(s)/M is the fraction of all students who attend school s ; 

= [P (1), .... p„eR-i)] 

is the vector of (R-1^ response option proportions for students in school s, 
and 

S 

R = {diag (P) - PP'^}'^ 1 <|)(s)(Pg - P)(Pg - P)' 

s=l 



is the matrix analogue of the intra-cluster correlation coefficient with P 
denoting the universe level vector of (R-1) response option proportions. 
With this partitioning one can show that the eigenvalues of 

Vp(SRS)"^Vp(DES) = [I + (m-l)R] 

have the form" 

V = [1 + (m-1) p^] ■ 

where p is the a-th largest eigenvalue of the intracluster corr&lation 
a "~ 

matrix R. Rao and Scott call these p quantities generalized measures of 

a 

homogeneity, analogous to the intracluster correlation p. For the simple 
goodness-of-f it hypothesis H^: P = , the simple random sample (SRS) 

can be written as 

. ^SRS ^'Pq) =^V^''^^""'^Poa] 

a = l 

• • 

where the are single degree of freedom central Chi-square variables, 
a 

For a portable value of p^ useful for modeling \^ = [l+(m-l)pj in compar- 
able samples with differing cluster sizes m, one could use 

P. = (\.-l)/(m-l) 

O 

R — 1^ 

= {{ 1 [l-P(r)]DEFF(r)/(R-l)]-l]/(m-l). 
r=l 

An extension of this self-weighting, two stage, with replacement model for 
\^ to a NAEP type design with effects for unequal weighting, stratifica- 
tion, and clustering is presented in the following section. These results 
are used to display the effect of sample design features on SRS based Wald 
Statistics for testing the fit of linear models 

H : G(P) = Xp. 

lO ' 
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and lineat hypotheses regarding "the model parameters p; that is, 
■ ' H : Cp = *. 

0 

. . . . \ ^ 

2 . 3 Design Effect Models for P-Value Covariance Matrices . ' " 

To develop a design effect model for the covariance matrix of a vector 

P of D* National Assessment domain P-values, we consider a three stage 

design with n county sized primary sampling units selected from a universe 

of N such,, units where the random frequency of selection for primary frame 

unit PFU(£) is n(£) . These n(£) are akin to the random selection^ 

indicators for without replacement samples where A.(£) is 1 when PFU(£) 

belongs to the sample and zero otherwise . The n(£) are allowed to assume 

values greater than 1 so as to accomodate so-called self-representing or 

certainty units. Following Chromy (1979), one can use th'kse random selec- 

tion frequencies to characterize a class of probability proportional to 

size (PPSO select^ion schemes including PPS with replacement, PPS without 

replacement, and PPS minimum replacement (PMR) . The PPS nature of these 

selection schemes implies that 

E{n(£)} = En(£) = ns(£)/s(+) 
. ^ = n(t)(£) 

where s(£) is a size measure known for each primary frame unit PFU(£) and 

N 

s(+) = I s(£) 
£=1 

is the universe level aggregate size measure. For the NAEP design, the 
size measure s(£) is typically the estimated PFU enrollment for the 13-year- 
old target .population. For with replacement selections, the n(£) are 
multinomial frequencies with 
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when HfS.'. For without replacement PPS designs n(£) = , the zero-one 

selection indicator, and 

E{n(il)n(ii' )} = 7l(M') 

■ . • ^ ^ ' ■ 

with 7t(£i!') denoting the joint inclusion probability for the frame units 
PFU(£) and PFU(£'). For Chromy's probability ^minimum replacement (PMR) 
design . . 

|n(£)-En(£) r=|n(£)-n(t)(£) I < 1. 

Specifically, for PMR designs 

Pr{n(£) = Int[n(t)(£)]+1] = Frac [n(t)(£)] 

and 

Pr{n(£) = Int[n(t)(£)]] = 1-Frac [n(t)(£)] 

where Int(x) denotes the integer part of x and Frac(x) depicts the frac- 
tional part of X. The PMR feature allows multiple selection from certainty 
units with n(t)(£)>l such that the number of hits n(£) is derived by randomly 
rounding the En(£) = n(t)(J^) proportional to size allocations up or down. 
The sampling variance function for this class of selection schemes has been 
parameterized in terms of variance <ad covariance components by Folsom 
(1980) utilizing double draw probabilities 

E{n(£)[n(£)-l]]/n(n-l) if 

^ (t)(M') = 

E{n(£)n(£' )]/n(n-l) if . 
These double draw probabilities (t)(M') arid the associated single draw 
probabilities (t)(J^) were dervied by Folsom as the expectations of single 
draw sampling unit indicators ^^(i) that take the value 1 when n(£)>0 and 
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selected primary frame unit PFU(£) is randomly assigned primary sampling 
unit (PSU) label i with i ranging from 1 to n; otherwise = 0- With 

the labels assigned as a random permutation of the digits 1, . . . , n, one 
can show that , 



0 E{\^(i)} = (t)(£) for all i 



and 



E{\^f'-J\^,(i')} = (t)(J^J^') for all^ij^ y 

.where expectation is taken over repeated samples and repeated random PSU 
label assignments. The single draw\ indicators have^the additional proper- 
ties 0 



N 

1 Kii) = 1 
£=1 ^ 



for all i, and 



1 = n(£) 

£=1 ^ 



These results lead to the following (!>(£) and (!>(££') summation identities: 

N . 
1 ■ (t)(£) = 1 
£=1 

and : 

_N N 
. I (!>(££•) = E{X(,,(i) 1 
£=1 o ^ £=1 ^ 

= (!>(£')- 

In the following subsection, the single draw indicators are used to define 
unbiased single draw variates in terms of the following single PFU ratio 
type estimators 
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y(£) = s(+) Y(£')/s(£) 
= Y(£)/(j)(£) 

where 

Y(£)^'= [Y^^ . ., Y^^(£), .... Yj3(j^_^)(£)] 

denotes a row vector of D(R-l) PFU totals and -the superscript T denotes 
matrix transposition. In the illustration above, will denote the 

(dr)-th element of the vector Y(£) specifying the number of age eligible 
domain d students attending school in PFU(£). who would give response option 
r to a particular NAEP item. 

2.3.1 Single Stage Covariance Matrix Models — — - — 

To develop a design effect representatiori^or the covariance matrix of 
a three-stage design statistic we begin by developing single stage results. 
Extending Folsom's (1980) single stage results to vector valued statistics, 
we consideir" the corresponding vector valued single draw variate 

^ N 

v(i) = 1 y(£), 
£=1 ^ 

with x(^) denoting the single PFU ratio type esTiimator Y(£)/(t)(£) defined 
previously.^ 

Now, one can show that each single draw sample variate y(i) is an 
unbiased estimator of the universe total'^Y(+); that is, 

N 

E{y(i)} = 1 E{\/i)} y(£) 
~ £=1 ^ ~ 

N ' - 

= ^ Y(£) 

£=1 • - 

• = Y(+) for all i. 
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Si.,:!-:,larly , the covariance matrix for each y(i) is 



N 



E{[y(i)-Y(+)][y;i)-Y(+)]'M = E{ I \;(i)[y(£)-Y(+)]} { I , (i) [y(A' )-Y(+) 1 } 

£=1 ~ ~ £'=1 * ~ ~ 



N • J 

£=1 * 



since 



X^ii) when £ = £' 

\g(i) \g, (i) = 
^ ^ 0 when Z f SL' . 

Taking the expectation over repeated samples and PSU label assignments (E) 
inside the summation, one obtains 



I (PSU) = I (|)(£)[x(J^)-Y(+)][x(J<^)-Y(+)]^- . 
^ £=1 

The cross-covariance matrix between y(i) and y(i') is derived similarly as 



N N T 
ZR (PSU) = E{ I I \o(i)\o,(i')[x(J<^)-Y(+)]lx(^')-X(+)M 
y £=1 £'=1 ^ ^ 



N N - ™ 

= I (!)(££• )[x(Jl^)-Y(+)] (X(^')-Y(+) ] 

£=1 £'=1 



for all i^i'. The fact that the single drawn sample variates have common 
covariance and cross-covariance matrices, provides a simple classical 
■ derivation of the variance for the mean of the single draw variate vectors 



n 

X= I,x(i)/n 
i=l 



N n 

= I [ I \ (i)]Y(£)/n(t)U) 

Z=l i=l ' , 

H 

= I n.(£)Y(Ji)/En(£). 

£=1 ~ •■ 

Notice that in the recast version y becomes the standard Hansen-Hurwitz 
(1943) estimator for a PPS with replacement selection. For a without 
replacement PPS sample with En(£)=n(£), ^ is the Horvitz-Thompson (1952) 
estimator. For the intermediate PMR designs, ;^ is Chromy^s (1979) esti- 
mator. In terms of the common covariance and cross-covariance matrices 
I^(PSU) and rR^(PSU), the covariance matrix for is 

V_(PSU) = I (PSU)/n + (n-l)IR (PSU)/n 
y ^ ^ , • 



= ly(PSU) [I+(n-l)Ry(PSU)]/n 



with 



Ry(PSU) ■= Iy(PSU)~-^ .IRy(PSU) 
defining the cross-correlation matrix and I denoting the D(R-l) by D(R-l) 
identity matrix. 

The following alternative expressions for I^CPSU) and IR^CPSU) make 
it easy to see that the form for V_(PSU) developed above is equivalent to 

y " . ■ ■ 

the Yates-Grundy. (1953) type, variance expression presented in Chromy for 
this general class of designs: 



N N 

I (PSU) = I I (|)(£)(|)(£')[x(J^)-X(J<^')][x(J^)-X(J^'Xr/2 
y £=1 £'^£ 

I 

a 

and 

S ' N N' , . . T ■ 

IR(PSU)=-I I [(|)(£)(|)(£' )-(!>(££•)] [X(J^)-X(J^' )] [XU)-X(J^' )] /2 • 
• ' ^ £5=1 £'^JZ . - 
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Weighting these component matrices together as indicated in V_(PSU) leads to , 

• y 

N N T 

V (PSU) =11 [(t)(il)(t)(il')-(n-l)(|)(M')/n][x(il)-X(^'>'itX(^)-X(^')l 
y £=1 I'fSi 

N N Y(£) Y(il') Y(£) ' Y(£') ^ 

V.(PSU) =11' [EnU)En(£')-EnU)n(£')] { - g^^jry} " e^^^ 

y £=r£'^£ 

When tEe PSU selection scheme gives all pairs of primary frame units S. and . 
£' a chance of being represented in the sample so that En(£)n(£') > 0 for 
all Si t the alternative component matrix expressions suggest unbiased 

estimators 



M T 
1 (PSU) =11 [n(£)n(£»)/En(£)n(£')](t)(ii)<t)(ii')6(M')6(M') /2 



and 



N T 

IR (PSU) =11 [n(£)n(£')/En(£)n(£')][(t)(ii)(Kii')-((»(M')]6(M')6(M') /2 

with- , ^ * 

6(M') = [x(ii)-xU')]- 
For a multi-stage sample such as the NAEP design, the PFU vector totals 
X(J^) imbeded in the definition of our 6(M') quantities must be estimated 
ba,sed on second and subsequent stages of sampling. The unbiased estimation 
of stage specific component matrices is complicated by this process. The 
following section develops the three stage covariance matrix model for the 
vector valued total estimator Y>(+) . ' 

2.3.2 Three-Stage Covariance Matrix Models for Estimated Totals 
For a three-stage NAEP type design where c schools are selected for a 
given package in each sample PSU and m* students are selected for^ package 
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assignment in each sample school, „ random selection frequencies ^(lls) and 



m(£st) characterize the number of selections of school s in PFU(£) and the 
number of selections of^ student t in school (J^s). For NAEP sample designs, 
the school and student level selections are without replacement. The 
school selections are. made with probability proportional \o estimated age 
class enrollment, say A(£s) J .that is ' 

, . c • 

E{|(i>s)|n(il)=l} = c A(ils)/A(il+) , ' 

= c (|)(sU). 

When multiple hits are allowed on the primary frame units, n(£) > 1 inde- 



npendenE" replicated samples of c schools are drawn from PFU(£) . 

The second stage conditional double draw probabilities are defined as 

^ . o . 

E{4(£s)[4(ils)-r]}/c(c-l) if s=s' 
E{\ (ij)\ (ij')|X (i)=l} = ... 

^ E{4(£s)4Us')}/c(c-l) if s^s- 

= <|) (ss'U) 

where ^£g(ij) is 1 if 4(£s)>0 and school s in PFU(£) is randomly assigned 
sample school label (ij) given \^(i)=l. The third stage sample is a^ simple 
random selection without. replacement so that 
E{m(£st)|4(£s) > 0} = m/M(£s) 

= m(t)(tUs) 

with equal single draw probabilities (})(t|£s) = M(£s) ^ where M(£s) denotes 
the number of age eligible students in school list anit SCH(£s) . The 
conditional double draw probabilities at the third stage are ^ 

E{^^gt(ijk)\^^^,(ijkV)|\^(i)\^^(ij) =1] = l/M(£s)[M(iisM] if tfV 

T (t>(tt^ Us) ^ 
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-and otherwise (J)(tt|£s) = 0 since for without replacement, selections 
\q (ijk) \. (ijk*) = 0; that is, the student list unit t can not be 
labeled botb sample student k and k' since student list unit t can be 
selected only once. With these definitions, a three stage single draw 
variate is defined as - 

N S(£) M(£s) . .. ' 

y(ijk) = I .1 1 a„ . (ijk) Y(£st)/(|)(£st) 
£=1 s=l t=l '^^^ 

with the three stage single draw indicator a^^g^Cijk) defined as the product 
of the stagewise indicators ^ 

The corresponding three stage single draw probability is defined analogous- 
ly; that is, 

E{of (ijk)} = (|>(£st) = (t>(£) (|>(s|£) (frCtUs). - 
£st 

With these definitions, it is ndt difficult to see that 

N S(£) M(£s) 
^ E {X(ijk)} =11 1 I (£st) 
£=1 s=l t=l 

= Y(+++) . 

for all sample students STU(ijk) where Y(+++) is the universe total of the 
response Vector Y(£st) with D(R-l) elements of the form 

Y^^Ust) = X^(£st) Y^(£st) 
where X^(£st) takes the value one when student list unit SLU(£st) belongs 
to subpopulation domain d and zero otherwise. The covariance matrices" for 
these thrive stage single draw variables can be derived simply using condi- 
tional expectations. For example, consider . \ 
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E{[x(ijk) - Y(+++)] (x(ijk) - Y(+++)]^} = 1^ (PSU) + Iy=,(SeH) + Z^CSTU) 

= Cov [x(ijk)] , 

where ' . ... „ 



N S (£) M(£s) 

I (STU) =11 (|)(£s) I (|)(tUs-j [yiUst) - y(£sj] [y(£st) - y(£sj]' 
y •. £=1 s=l t=l . ~ ~ ~ ~ 

with the school list unit mean y(£s.) defined as 



M(£s) 

X(ils J = I (|)(t|ils) x(ilst) 

t=l • " 

= Y (£s+) / ^U) (|>(sU) . ■ 

= Y (£s+) / (|)(£s) . 

For NAEP type designs with simple random selections at the third .stage one 
^obtains the simplified form ^ 



2 N S(£) 
(STU) = M(..) ^I^ ^I^ [ ^ ] I^Us) 



where e(£s) = M(£s)/M(++) denotes the school list unit (£s) fraction of the 
tbtal student population count M(++) . The covariance matrix for school, 
list unit (£s) is 

(£s) = [diag {nUs)} - nUs) n(ils)^] 



with 



M(£s) 

7t(£s)^ = I ^ Y(£st)^/M(£s) 
t=l 



denoting ' the/vector of 'D'(R-l) .cell ^^proportions for school\list iii^it (Jis)/ 
The dr-th cell in our example denotes membership in domain d and item 
response group r. * ' 
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The school stage covariance matrix component has the^Toxm 



N Sis.) X 

1 (SCH) ^ Ml) ^ ^(su) [x(is.)-xu^Ji ^yii--)J 

y ji-i .3=1 ~ ~ . . 



with 

[X(As.) - ziH..)] = {ias+)/HS.) <t>(sU) - Y(£++)/(t)(il)} 

where 9(£) = M(£+)/M(++) denotes the fraction of student age eligibles in 
PFU(£). If the relative size measure 

(t)(£s) = [S(£)/S(+)] [A(£s)/A(il+)] 
= M(£s)/M(++), 
then the sample „ is self -weighting since 

<t»(£st) = (t)(£s)/M(£s) = 1/M(++) 
and the total inclusion probability for student list unit SLU (£st) is^ 

7T(£st) = ncm (t)(£st) = ncm/M(++), 
a constant for all nsm sample students. Inthis simplified case 

2 N M(ils) T 

I (STU) = {!(++) I I QiSs) {diag [nUs)] - n(£s) n(£s) } 

y ' £=1 s=l . ~ 

and 

. 2 ' N SU) ' . T • 

I (SCH) = M(++) I I e(ils) (n(£s) - n(£)] [n(£s) - niSL)] . 

y £=1 s=l 

The general form of the PSU level covariance component matrix is 

• ' • N 1 

i (PSU) = I (!>(£) [y(J^..) - y(...)] [ y(^..) - y(...)] 

y ~ " . ~ ~ ~ . . 



with. 



,iyu.;) - y(..,)] = ^.^^'^ - 2 ^ • 
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For a self-weighting sample 

2 • N 

I (PSU) = M(++) 1 e(£) [nU)-n] [7t(£) - n]\ 
' ' £=1 

In general, one notes that 

Cov(x(ijk)] = (PSU) + (SCH) + 1^ (STU) 

N S (£) M(£s ) 

£=1 s=l t=l . . 

N S (£) M(£s ) 

= • Z I I Y(£st) Y(£st)'^/(t)(£st) - Y(+++) Y(+++)'^ 

£=1 s=l t=l ^ " ^ 

N S(£) M(£s) 

= M^(++) { I 1 1 Y(£st) Y(£st)'^/M(++)^ (J)(£st) - n n^} 
£=1 s=l t=l ^ ' 



= I^(TOT). 

y 

When the sample is self-weighting with 

4(£st) = 1/M(++), 
tfie common cov3riance matrix for each ;^(ijk) is 



(TOT) = M(++) {diag [n] ' nn^}. 
Notice that in the self-weighting case, Iy(TOT)/M(++)'^ is the SRS with 
replacement multinomial covariahce component matrix. 

Various cross-covariance components can be define for the three-stage 
single draw variables. These cross-covariance components are derived as 
follows: " » 



Gov [xUjk); x(ijk')] = Gov [y(i..)] +E Gov IxCij.)]* 
' • ' PSU ' - PSU SGH 

• . ' . ' + E e' Coy . /[x(ijW)rx(ijk^X] ' ' Y 

^ PSU SGH STU'. ; . 

' ' ' = Zy (PSU) + (SCH) + IRy (STU) 
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where 

y(ij.) = E {y(ijk)|PSU, SCH} 
STU ~ 

denotes the conditional expectation ofvy(ijk) over repeated student selec- 
tions and label assignments given the PSU and school selections and label 
assignments. The conditional expectation over* school and student selection 
and label assignment of X^ijk) given the PSU selection and labelling is 
similarly denoted by 

X(i..) = E E {y(ijk|PSU}.. 

SCH STU ~ - , 

The matrices (PSU) and 1^ (SCH) were defined previously, and 

N S(il) M(£sl M(£s) rr, 

IR (STU) =11 (t)(ils) I I 6(£st)6(£st)VM(£s)[M(ils)-l] 
y £=1 s=l ■ t=l t'=l 

with ^ „ 

6(£st) = [xiSLst) - .x(J^s.)] . • 

= M(ils) [Y(ilst) - ^n(ils).]/(t)(ils). 

With further manipulation the following form for the between ktudent within 

school Cross-covariance matrix is obtained 

IR (STU) = -M(++) I I. ( TriTT^ ] 2^ (^s)/[M(£s)-l] 

y £=1 s=i ^^^^ ^ . ■ V 

recalling that 

Iy(£s) = {diag IniSLs)] - n(ils) 7T(ils)^} . 
Turning to the between school within PSU cross -covajriance matrix one obtains 

.Gov [y(iik)V. y(ij'k')] = Cbv + E„ Cov [x(i'j. ) ; .X^ij ' J • 

'PSU PSU SCH 



= I (PSU)" + IR (SCH) • 

- y y 



with 



N S(£) SiSL) T 

IR (SCH) = I (t)(£) I I (t)(ss'|Jli) 6(£s3. 6(£s') 
y SL=1 s=l s'=l 
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where 

Finally, the between PSU cross-covariance matrix is defined as follows 

N „, 
Cov [x(ijk); zii'yk')] = 1 (|)(££' )' 6(£) 6(£')' 

£=1 . 



= ZR (PSU) 

y 



where 



Armed with the covariance and cross-covariance component definitions 
specified above, one can derive the covariance matrix for 



n c m . • 

i=l j = i k=l - - 

the three-stage analogue of the general class of with replacement and PMR 
single stage estimators. The covariance matrix for ^ is ^ 

Cov (£] = I,^(PSU) [I + (n-1) Ry(PSU)]/n + r(SCH) [I + (c-1) Ry(SCH)]/nc 
+ (3TU) [I +_ (m"-l) (STU)]/ncm. 
A design effect version of this model can be formed as follows: 

Covlx] = {Iy,(SRS)/ncm} W {I +. (ncm-1) A^CPSU) R^CPSU) 

+ (cm-1) [A^(PSU) {I - Ry(PSU)} + A^CSCH) Ry(SCH)] 



where 



+ (m-1) [Ay(SCH) {I - Ry(SCH)} + A^CSTU). RyCSTU)].} 



W = ly(SRS)"-^ ly(TOT) 



= The uritiual weighting effect; 
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Ay(PSU) = Zy(TOT)"^ ly(PSU) 

= The within PSU clustering effect; 
_i 

Ry(PSU) = ly(PSU) IRy(PSU) 

s The PSU stratification and PMR selection effect; 
Ay(SCH) = ly(TOT)"-^ ly(SCH) , 

s The within school clustering effect; 
Ry(SCK) = Zy(SCH) IRy(SCH) ^ 

s The school stratification and PMR selection effect; 
A (STU) = 1 (TOT)"-^ 1 (STU) 

= (I - Ay(^SU) - Ay(SCH)] 

= Between student within school frp.ction of total variation; 
and . 
Ry(STU) ='Iy(STU)"'^ 2Ry(STU) 

= The effect of without replacement student selection. 
As suggested above, the cross-covariance matrices account for both the 
effects of minimum replacement (PMR) or without replacement selections and 
stratification. The effect of explicit and implicit stratification is 
expressed through the sampling expectation of selection frequency products, 
E n(£) n(£.'), and associated double draw probabilities , (|)(£J?/). Recalling 
the YaLes-Grundy form of the PSU level variance function 

, V (PSU) =1 1 [En(£)En(£') - En(£)n(£')] d(££') d(££')^/2 
1 y £=1 Z^n 

with 

4(££');= {Y(£)/En(£) - •¥(£' )/En(£0} , ; . 

it is clear that explicit stratification ^ would imply the Sadjspendence of 
^selection frequencies n(£) and tiCZ') for.PFU^s in different strata. This 
independence would cause the between PSU contrasts d(££* ) in V--(PSU) to 




drop out since En(£)n(£' ) = En(£) En(£'). Therefore, while the V-(PSU) 
variance expression and the variance-covarianc.e component analogue are not 
written in the familiar stratified form, they reduce to such a form when 
the [En(£) En(£*) - En(£)n(£')] coefficients are set, to zero for the be- 
tween stratum terms. With Chromy's sequential PMR zone selection scheme, 
implicit stratification effects are achieved by purposively ordering the 
frame listing so that proximate units are expected to be more alike than 
distant units in terms of the survey outcome measures. The reduction in 
sampling variance associated with the effect of implicit stratification is 
reflected in the Yates-Grundy variance form by a tendency 'for the cpeffi- 
cients 

[En(£) En(£*) - En(£)n(£^)] 
to • approach zero as the distance between frame units £ and £^ increases. 
The variance-covariance component representation for V-(PSU) displays the 
combined effect of minimum replacement selection and implicit stratifica- 
tion in the form [1 + (n-1) R^(PSIJ)]. For a scalar statistic this express- 
ion reduces to [1 + (n-l)p^(PSU) ] where p^(PSU), the common correlation 
among the n single draw variates y(i), is expected to be increasingly 
negative as the efficacy of the implicit stratification improves. 

2.3.3 Design Effect Model for the NAEP P-Value Covariance Matrix 
The design effect model developed for the linear statistic the 
vector of estimated domain by item response category totals Y(dr), can be 
extended to the vector/p of D(R-l) response category proportions considered 
previously by applying' the Taylor Series linearization technique implicit 
in the section »2: 2 treatment of generalized design \,e£fect methods.^ Begin 
by letting ' * * ^ 
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• R 

[Y(+++)] = Y(dr)/ I Y(dr) = P(dr) 
dr 1 

• - r=l 

and let denote the 1 x D(R-l) vector with elements 

h, (uv) = 8P(dr)/8Y(uv) . 
dr 

The elements of H, have the following form 
'^dr 

[1 - P(dr)]/X(d) if u = d and V = r 
H , = - P(dr)/X(d) if u = d and V ^ r 

^ ^ 0 otherwise 



where 



R 

X(d) = 1 Y(dr) . 
r=l 



Letting h'^ = (Hii, . . . , hJ^, . . . , hJ(r.i)) defining the matrix of 
partial derivatives of P with respect to the elements of Y(+++) where the T 
superscript denotes matrix transposition, then 

Vp(DES) = H Cov[x] . 
The expression for the design based covariance matrix of P stated above can 
now b^e used along^ with the three-stage component represantation for Gov [^1 
to produce an analogous component representation for Vp(DES). This is 
accomplished by defining analogous covariance and cross-variance matrices 
for each stage as. follows: ^ ^ 

oIp(STAGE)' = H ly'(STAGE) 

^and T ^ 

ZRp(STAGE) = H IR^CSTAGE) H ' 

with STAGE assuming the PSU, SCH, and STU levels for the NAEP design. 

An alternative form of the Taylor series linearization that provides 

explicit definitions for the component matrices is. to define linearized 

. ' • • .... ■ ' .r V. 

• A' • . .... , C . • . ^ . 

variates . ^ \ . . . . ' * « , ' \. 
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\ ; z(ijk) = H x(ijk) 

N SU) M(£s) 
=11 I a. . (ijk) z(£st) 
ii=l s=l t=l >ist . 

where 

z(£st) =: H x(^st) 

= H Y(£st)/(t)(£st) 

0 

= M(£s) Z(£st)/(t)(£)(t)(s,£) . • ^ - 

The Z(£st) = HY(£st) vectors defined above have elements of the form 

Z^^(£st) = X^(£st) [Y^(£st) - P(dr)]/X(d) 
recalling that X^(£st) is the one-zero indicator for domain d membership 
and Y^(£st) is the one-zero indicator for response category r. Using the 
linearized three-stage single draw variates z(£st) in place of the ^(JZst) 
vectors in the ) and, IR^C ) definitions yields ) and ) 



matrices suchvthat 

STAGED = 1 rSTAGE^ = H l^. 



Ip( STAGE) = (STAGE) = H I (STAGE) 



and 

IRp(STAGE) = IR^(STAGE) ^ H ly(STAGE) . 

In terms of these quantities, the school level population mean vectors 

M(jZs) ' 
z(£sj = I z(£st)/M(£s) 
* t=l . 

have elements . 

z^^(£sj = X^(£s+)[P^^(£s) - P(dr)]/X(d) (t)(iis) 

= e^(£s) [P^^^(iis) - P(dr)]/e(iis) 

where P^^(£^) is the proportion of domain d members of school list unit 

(£s) tha,t would give item response option -r and ,9^(£s) is the fraction of 

all domain d .members .at.tenlding schpol in school list unit (J^sj. the PSU 

level mean vector 
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z(S. ) = I (|>(s|£)z(£s..) 
~ ■ * * s=l 

o 

has elements ' • ■ 

a ) = X^(£++) (P, (Ji) - P(dr)]/X(d)(t>(il) 
dr • • d ar 

= e^(£) [P^j^Cii) - P(dr)]/(t)(£) . 
If one lets D^(£s) denote a D(R-l) by D(R-l) diagonal matrix with the 
(dr)-th element C^j^C^s) = e^(£s)/(t)(£s) , then the school level mean vector 
of linearized variates is 

zCAs.) D^(ils) (P(£s) -. P] . 
'Defining D^(£) with (dr)-th element t^^W = 6^(£)/(|)(£) , a similar form for 
the PSU le,vel mean vector is obtained, namely 

2(£..) = D^(£)(P(il)-P] . • ' 

When no members of domain d attend school list unit SCH(£s), then 6^(£s) = 0 
and P^ (£s) = 0. Similarly, if no members of domain d attend school in 
primary frame unit„PFU(£), then 6^(2) = 0 and ^^^W = 0. 

In terms of these linearized variate vectors, the stage specific 
covariance matrices have the form 



N SiSL) M(ils) , X 

I (STU)..= 1 1 ^iSLs) 1 (z(£st)-z(ils.)] [z (£st)-2 (ils . ) ] /M(£s) 
£=1 s=l t=l 

N S(£) M(ils) „ ° . , ' 

= 11 <^(.S.s) 1- 2(ilSt)2(ilSt) /M(ils) 
£=1 s=l t=l ~ 

N S(£) ■ 
-11 (|)(£s)z(£sjz(£s.) . 

£=1 s=i ' ~ ' ; ^ : 

Letting D^CAst) dte'noti; a D(R-l) by D('R-l) ' diagonal matrix With elements. 

• " ; '• . ' ' • ■ . , - ' '• • 

e^fist) = X ,(£s't)/X(d) , the between s,tud'ent coVariance matrix becomes . 
d d 



37 

-31-, 



N Sm M(ils) c , T 

I„(STU) = I I I Do(£st) [Y(ilst)-P] [V(ilst)-P]^Do(ilst) M(£s)/(t)(£s) 
^/ ■ £=1 s=l t=l ^ ~ ~ ~ ~ e 

N S(£) 

- I I (t)(£s)D;.(ils)(P(£s)-P] [P(£s)-P]^n<.(ils) ^ 
£=1 s=l ^ ~ ~ ~ ' ~ 

where the (dr)-th element of Y(£st) is Y^(£st), the one-zero indicator for 

item response category r. The between school withi^n PSU covariance matrix 

is ' ' 

N S(£) „ 
I (SCH) = I Z (t)(£s)[z(£s )-z(£ )][z(£s )-z(£ )V 
£=1 s=l ~ . ~ ~ . ~ .. 

N S(£) ■ 
= 11 (t)(£s)z(£s.)z(£s 
£=1 s=l ~ ' ~ ' 

N 

- I (t)(£)z(£ )z(£ y . 
£=1 



Therefore 



N S(£) 

- Ip(SCH) = I. I (t)(£s)D<.(£s) [P(£s)-P] [P(£s)-P] D<.(£s) 
£=1 s=l ~ ~ ~ ~ 



N 



- I (t)(£)D.(£)[P(£)-P] [P(£)-P]'D.(£) . 
£=1 t ~ ~ . ~ i> . 



The between PSU covariance matrix is 

N 

I (PSU) = I (t)(£) [z(£..)-z(...)] [z(£..)-z(...)]^ 

= 1 (t)(£) z(£..)z(£..)^ 

■ £=1 ■ ' ' 

since . . 

» , "..{." 

, ■ .. ' ^ 

N ■■ ' ■ ■ , ■ ' 

/ ■ z(._) = I (t)(£) zU..) 
■'.£=!■, 



38 



-32- 



N 

= 1 ^^(S.) [P(£) - P] = 0 . 

Therefore, with the (|> weighted mean of the linearized vectors z equivalent 
to the null vector, the between PSU covariance maCrix can^be^Witten as 

u 

. N T . . 

ip(PSU) = I (^WDfU) 1P(£)-P] [P(J^)-P] DfC^) • 
^ £=1 ^ b 0 • 

^Combining these results, one obtains the following expression for the- total 

covariance matrix 

Gov [z(ijk)] = Ip(TOT) 

= Ip(PSU) + IpCSCHO + Ip(STU) 

N S(£) M(£s) >' T 
= 11 I D (ilst)[Y(£st)-P] [Y(£st)-P]'DgUat)M(iis)/(j)(ils) 
£=1 s=l t=l ~ ~ ~ 

Letting W(£st) = M(£s)/ [ncIn(t>(J^s) ] denote the sample weight or inverse 
selection probability for student (J^st) , one can recast Ip(TOT) as a block 
diagonal matrix with blocks of the form ^ 

N S(£) M(£s) T 
iJ(TOT) =■ ncm {I I I W(ilst)X (ilst) [Y(iist)-P(d) ] [Y(£st) -P(d) ] /X(d)2} 
^ £=1 s=l t=l 



= ncm W(d)f(d){diag [PW(d) ] -PW(d)P(d)'''TP(d)PW(d)^+P(d)P(d)'^]/Em(d) 



where 



' ' ' N • S(£) M(£s) 

W(d)" =1^1 1 W(£st)X (£st)/X(d> 
' £=1 s=l , t=l • ' ■ ■ , 

is' the average weight for all domain d members in the universe -and 

• f(d) = Em(d')/X(d) 
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is the expected sampling fraction for domain d with 

N S(£) M(£s) 



Em(d)= ,I 1 1 1ncm (|>(£s)/M(£s)]X (£st) 
£=1 s=l t=l 

0 

N S(£) M(£s) 
= 11 1 W(£st)". X (£st) 
. £=1 s=l t=l 



denoting the expected domain d sample size over repeated samples . The 
PW(d) vectors represent the weighted universe level response option distri- 
bution for domain d members; that is 

N S(£) M(£s)^' 
PW(d) =11 1 W(£st) X (£st)Y(£st)/W (+++) 
£=1 s=l . t=l a . 

with W^(-»-++) denoting the universe level weight sum for domain d members. 

For a self-weighting sample with common weight 

W(£st) = M(£s)/ncm (t)(£s) 
= M(++)/ncm 



one observes that 



and 



W(d) = M(++)/ncm 
f(d) = ncm/M(++) 

PW(d) '= P(d) . 



For a self-weighting design one note^'s therefore that Ip(TOTX/nsm defined on 
page 34 assumes the with replacement simple random .sampling multinomial, 
"'form; that is c- » , 

• Ip(T0T)7ncm = {diag [P(d)] - P(d)P(d.)^}/Em(d) ,^ 
with the expected' domain sample size specified as 

Em(a) = ncm X(d)/M(++) 
= ncm n(d) . 
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For the typical multi-stag^ PPS sample design utilizing approximate size 
mea sures , the unequal weighting effect is defined as : 



" = VpCSRS)"-^ Ip(TOT)/ncm . 

The matrix u)p is .block diagonal with blocks 

iL (d) = W(d) f(d")|ciiag [P(d)]-P(d)P(d)^]"^ { diag [PW(d)]-PW(d)P(d) 
- P(d)PW(d)^ + P(d)P(d)^ } \ . 

When a correct-incorrect dichotomous response distribution is considered, 
the ddmain d effect of unequal weighting uip(d) can be recast in the follow- 

0 

ing form 

JipCd) = W(d)f (d) (PW(d)-2PW(d)P(d)+P(d)2]/P(d)[l-P(d)] 

= W(d)f (d) {[PW(d)/P(d)] + [l-PW(d)]/[l-P(d)] - 1} . 
Recalling' that f (d) is the siibpopulation d mean of the inclusion probabili- 
ties 7t(£st) = W(£st)'^, the product of the average weight W(d) and the 
expected domain d sampling fraction f (d) can be written as 



W(d) f(d) = W(d) ^ Wjj(d) 



where 



Wjj(d> = 1/f (d) 

N S(£) M(£s) , 
'=(11 I W(£st)"^ X (£st)/X(d)] 
£=1 s=l t=l 

is the universe level, harmonic mean of the W(£st) weights for domain d 
members. Noting that the W(£st) weights are nonnegative quantities, it is 
clear that W(d)f(d) ^ 1 since the harmonic mean W(d) is less than the 
arithmetic mean W(d) for nonnegative variables- 

2.3.4 Estimates for Composite Covariance Matrix Components 
To produce estimates for the three-stage covariance matrix components 
defined . in the previous sectio'n, one can begin by building a consistent 
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estimator for Ip(TOT). Note that 



, ( Z Z Z X^(ijk) W(ijk)/n(ijk) ] 

^(d) = j=l 

.. n c m 

[ Z Z Z- X (ijk) W(ijk) ], 
i=l j=i k=l ° 

is a consistent ratio estimator for W(d) and that 

: ^ ^ n c m 

f(d) = m(d) V [ Z Z Z, X ((ijk) W(ijk) ] 
i=l j=l k=l 

is similarly a consistent estimator for f(d). Therefore 



[ Z _Z Z X^tijk) W(ijk)2 ] 
W(d)f(d) = m(d) .{ i-1 j-1 k=l 



n * c m ' 
[ Z Z Z X,(ijkj W(ijk) V 

i=l j=l k=l ° 



is a consistent estimator for the associated unequal weighting factor. The 
consistent sample estimator W(d)f(d) is equivalent to the» unequal weighting 
design effect proposed by Kish (1965) and others. The derivation presented 
here shows that there is an additional term in the unequal weighting effect 
that . contrasts the universe level weighted mean of the correct response 

o 

indicator, PW(d), with the " subpopulation proportion correct P(d). This 

.additional unequal weighting factor 

Q(d) = {[PW(d)/P(d)] + [l-PW(d)]/[l-P(d)] - 1} 

is less than one when P(d) 0.5 and PW(d) > P(d). When PW(d) > P(d), this 
.implies that Wi(d)'> WqC^) where 'Wi(d) denotes the universe level mean of 

the weights for domain d members who respond correctly and WqC^) the 

corresponding mean for domain d members who respond incorrectly. Therefore 
•Q(d) < 1 when the harmonic mean of the inclusion probabilities for domain^ d 

c 

members who respond incorrectly is greater than the harmonic mean inclusion 



probability for those domain d members who would respond correctly; that is 
for P(d) S 0.5, Q(d) < 1 when the sample design overrepreserits domain d 



members who ""would respond incorrectly to the item. For items with 
P(d) < 0.5, 'Q(d) is less than one when PW(d) < P(d) which implies over*: 
representation of domain d members who would respond correctly. For the* 
NAEP desigrt where schools in low income inner city areas are- over- 
represented, there will be a tendency for overrepresentation of persons who 
would respond incorrectly. The effect of this overrepresentation on the 
Q(d) quantities should not be expected to counterbalance the rather substan- 
tial unequal weighting design effects. The total population value of Wf is 
around 1.35 for a single NAEP package sample. Consider for example an item 
with P = 0.55 and PW = 0.95, then Q = 0.838 and uip .= i.l3. Consistent 
sample estimates of the PW(d) can be formed using the squared weights 
W(ijk)^ to compute the weighted proportion giving response option r as 



PW(dr) = { 



n c 

Z ° I 1 WCijk)2 X,(ijk) Y (ijk) 

i=l j = l k=l \ 

n c m 

111 W(ijk)2 X (ijk) 
i=l j=l k=l . 



Utilizing these consistent estimators for W(d), f(d), and PW(dr), a con- 
sistent estimator for the domain d block of Ip(TOT)/ncm is 

Wp(TOT)/ncm = W(d)f(d) SWp(d)/m(d) 



where 



SW (d) = { diag [PW(d)] - PW(d)P(d)^ P(d)PW(d)^ + P(d)P(d)^.} . 



These considerations lead to the consistent estimator for Zp (TOT) 
Zp(TOT) = ncm BLK-DIAG {W(d)f (d)SWp(d)/m(d) } . 
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While one can produce Taylor-Series- approximations for the separate 
stagewise covariance and cross-covariance matrices Ip( STAGE) and ZRp( STAGE) , 



where. STAGE represents a generic design stage assuming the. levels PSU, SCH 
(school), and STU (student) for the NAEP design, such approximations re- 
quire the calculation of the PSU and school level double draw probabilities 
(!>(££*) and (|)(ss 'I Ji) . On the other hand, simple analysis of variance type 
estimators exist for the following composite component matrices 

Sp(STU) = Zp(STU) - IRp(STU) 

Sp(SCH) = Ip(SCH) - IRp(SCH) + IRp(STU) 

Sp(PSU) = Ip(PSU) - IRp(SCH) + IRp(STU). 

These composite component matrices are relatively easy to estimate and 
provide the necessary ingredients for parameterizing the following design 
effect version of the P-value covariance matrix model: . 

Vp(DES) = Vp(SRS) ilip [1 + (ncm-l)Rp(O)' + (cm-l)Rp(PSU) + (m-l)Rp(SCH)] 
where 

ip =,Vp(SRS)'^ Ip(TOT)/nsm 
Rp(0) = Ip(TOT)'^ 2Rp(PSU) 
■ Rp(PSU) = Ip(TOT)'^ Sp(PSU) 
Rp(SCH) = Ip(TOT)"' Sp(SCH) . 
The composite component definitions above also lead to the following useful 
identity 

Ip(TOT) = Ip(PSU) + Ip(SCH) + Ip(STU) 

= IRp(PSU) + Sp(PSU) + Sp(SCH) + Sp(STU) . 
This identity combined with the consistent estimator for Ip(TOT) and the 
Taylor Series ANOVA estimators for Sp(PSU), Sp(SCH), and Sp (STU) provide a 
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consistent estimate for the component ZRp(PSU) due to primary stratifi- 
cation and without replacement (or PMR) selection; that is, 

ZRp(PSU) = (Ip(TOT) - Sp(PSU) - Sp(SCH) - Sp(STU)] : 

The SpC ) matrices are estimated using the Taylor-Series linearized single 
draw variate vectors * 

'v. 

N S(£) M(£s) . , 

z(ijk) =11 I a„ .(ijk) z(£st) 
£=1 s=l t=:l , 

where the (dr)-th element of z(£st) has the form 

z^^(£st)-=M(£s) X^(£st) (Y^(£st)-P(dr)]/X(d) (l)(£s) 
= ncm W(£st) X^(£st) [Yj.(£st)-P(dr) ]/X(d) 
with P(dr) and X(d) denoting sample estimates for the corresponding popula- 
tion parameters. Recall that W(£st) = M(£s)/ncm (|)(£s) is the sample weight 
for student listing unit SLU(£st). In terms of the z(ijk) lirfearized 
single draw variate vectors, one computes the following ANOVA type matrix 
of mean squares and cross-products: ^ 

ncm 

^ MSp(STU) = - I I 1 [z(ijk)-z(ij J] (z(ijk)-2(ijj] /nc(m-l) 
i=l j=l k=l . ~ ~ 

with z(ij.) denoting the school level sample mean vector 



z(ij J = I z(ijk)/m . 
k=l 

The corresponding between school within PSU mean square matrix is 

n C ^ ^ rp 

MS„(SCH) =11 (z(ij.) - z(i..)][z(ij.) - z(i..)] / n(c-l) 
1=1 j-1 
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with. - . 

c 

z(i_) = 1 2(ij J/c . . 

The between PSU mean square matrix is 

MS (PSU) = 1 2(i_)|(i_)V(n-l) 
i=l 

noting that the overall mean of the z(ijk) vectors is the null vector. 

In tsrms of the single draw variate vector covariance and cross co- 
variance matrices defined previously, it is not difficult to show that 

E { MSp(STU)} = Sp(STU) = [Zp(STU)-IRp(STU) ] 
E {MSp(SCH)-MSp(STU)/m} = Sp(SCH) = [Zp(SCH)-aRp(SCH)+2Rp(STU) ] 

and 

E {MSp(PSU)-MSp(SCH)/c} = Sp(PSU) = [Zp(PSU)-IRp(PSU)+2Rp(SCH0 ] . 

The composite component model and associated component estimators for 
the P-value covariance matrix V'p(DES) have obvious extensions to the trans- 
formed P-value case. With H denoting the matrix of partial derivatives of 
G(P) with respect to the elements of P evaluated at P = P, then 



Iq(TOT) 


= H 


Ip(TOT)H 


Sq(STU) 


= H 


Sp(STU) H 


Sg(SCH) 


= H 


Sp(SCH) 


Sq(PSU) 


= H 


Sp(SCH) 



and >^ 

IRq(PSU) = Ig(TOT)-Sg(PSU)-Sg(SCH)-Sg(STU) . 

With the simple random s,ampling covariance matrix estimator for G(P) 

depicted by ' ^ ^ ^ 

Vg(SRS) = H Vp(SRS) , 
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the corresponding unequal weighting matrix ui^ and the composite correlation 



matrix analogues Rq(0), R^CPSU), and R^(SCH) have' the same form as the 
original P-value component matrix estimators with G replacing P in the 
defining equations. 

A straight forward extension of the Taylor Series linearization argu- 
ment applied to -G(P) provides an analogous model for the design based 
covariance matrix of 

g = (X^V^(SRS)''^X]"'^X^V^(SRS)''^G 
= M(SRS) G , 

the SRS based weighted least squares estimate for the G(P) = Xg linear 

model coefficients. While one might initially question the use of the SRS 

based covariance matrix in the definition of g above, recall that use of 

V-(DES) implies full knowledge of the design based P-vaJue' covariance 
G 

matrix V.(DES) which in turn provides for calculation of design based Wald 
G 

statistics; that is to say, if one use V^(DES) in the defini.tion of P then 
no extension of ^^o and Scott^s approximate methods are required. Returning 
to our SRS bas^^B, one can further consider a matrix of estimated contrasts 
eg. far as the first order Taylor Series linearization is concerned Cg 

is equivalent to a linear transformation of the original vector P of domain 
P-values ; that is 

Cg = [CM(SRS) H] P . 

ft 

The corresponding Taylor Series component estimators have the form 

Z^p(TOT.) = (cm) IpXTOT) (CM)^ 



■ 47 

-41- ^ 



' ^ ' ■ ■ . ' \. ........ ■ r • ■ . ■ : '' ^ 

S^p(STU) = (CM9)Sp(STU) (Cffli)^ .1 




s^p(scH) t= (cffii)Sp(scH) (cm) 


1 




and 






S^p(PSU) = (Cfffl)Sp(PSU) (Cffil)^ . 


I 




where M is shorthand notation, for M(SRS). For a NAEP style three-stage 


1 


a* 


design, the generalized design effect- matrix for Cp is therefore of the 
form 

DEFF(Cp) = {C[X V-(SRS) ^X] C'} {(eMH)V (DES)(CMH)M 


■ 
1 




= i^,p[I+(ncm-l) R^,p(0) + (cm-l) R^,p(PSU) + (m-l) Rgp(SCH)]. 


1 




2.3.5 Asymptotic Distribution of SRS based NAEP Wald Statistics 






The partitioning of the generalized design effect matrix for Cp devel- 






oped in the previous section leads to the following representation for the 


1 




asymptotic distribution of the SRS based Wald statistic 






A 

X^p,.(Cp) = 1 i [l+(ncm-l)p (0) + (cm-l)p (PSU) + (m-l)p (SCH)]x^ 
oX\0 ^ a a a a a 
a=l 


• 

1 




where the are independent single degree of freedom chi- square random 
a 






variables with coefficients defined in terms of the left and right hand 


1 




eigen vectors of DEFF(Cp), say L and R, and the component matrices uj^q, 






R^p(O), R^^p(PSU), and R^p(SCH). Specifically, if is the a-th row of L 






and r is the a-th column of" R, then the generalized effects of unequal 




0 


weighting, stratification, PMR selection, and clustering are defined as 




ERIC 
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- 03(0) = { [.<Il^^pX(ilx^J-J^ 



"a \ 
and ' 

, P3(SCH).= { £3 [i.cpRcp(SCH)] } V 
For uristratified PPS with replacement selections at each stage of sampling, 
the IRp( ) cross-variance component matrices are null so that the com- 
posite components Sp( ) .equal the corresponding . covariance components 
Ip( ). In this case simple single degree of freedom contrasts CP and more 
complex single degree of freedom contrasts Cg will have positive design 
effects of the form 

DEFF(Cp) = [l+(cm-l)p^p(PSU)+(m-l)p^p(SCH)] 



since the cluster correlations 

W 

must be nonnegative. This follows from the fact that the Ip(TOT) and 



(STAGE) = {(CMH) Ip(TOT) (CMH)^}"^ { (C^^HX Zp(STAGE) (CMH)^] 



I^(STAGE) matrices are all positive definite and 

Ip(TOT) = Ip(PSU) + Ip(SCH) + Ip(STU).. 

As indicated earlier, the iL matrix for the R = 2 correct-incorrect response 

pattern case is a D x D diagonal matrix with elements iD^^ taking the form. 

D ■ 
I q2jp(d)[i-p(d)]/m(d)} ^>p(d) 

\^ = ^ ^ — 

1 q2p(d)[i-p(d)]/m(d) 
d=l ^ " 

where denotes the d-th element of CMH. Notice that ui^^ is a weighted 

average of the u>p"(d) quantities with the weights q^P(d) [ l-P(d) ] /m(d) all 



positive and the ^(d) all expected to exceed 1. Therefore, one should 
expect U)^ Q to exceexLJL ^ i 

While one should therefore expect design effects for single degree of 
freedom P-value contrasts to exceed 1 for unstratified with replacement 
cluster samples, the tendency for the PSU and SCH (school) stage specific 
covariance matrices Ip( ) to have positive covariance terms for domains d 
and d* that are" typically represented in the same schools and PSUs will 
cause contrasts among c. such domain P-values to have smaller cluster correla- 
tions than observed for the separate domain P-values. Lepkowski and Landis 
(1980) examining data from the Health Examination Survey (HES) and a 1974 
University of Michigan Survey Research Center Omnibus (OMNI) Survey , ob- 
served this tendency for the DEFF of P-value contrasts to be substantially 
smaller than the DEFF of individual P-values. The size of the proportional 
reduction in an average contrast DEFF relative to the average P-value DEFF 
was 60 percent for the HES and 9 percent for the OMNI survey. The size of 
the proportional reduction factors observed by Lepkowski and Landis 
depended oh the magnitude of the average P-value DEFF. For HES where the 
P-value DEFF's averaged 3.91, an overall 60 percent reduction was observed. 
For the OMNI survey where the P-value DEFF*s averaged 1.10, the overall 
proportional reduction for cpntrast DEFF*s was only 9 percent. A similar, 
tendency for the proportional reduction to vary with the mean P-value DEFF 
was observed across dependent variates within the two surveys. 

For designs with stratification and without-ireplacement or PMR selec- 
tions at the various stages, the general expressions for the composite 
components contain cross-covariance matrices IRp(PSU) and ZRp(SCH) that are 
expected to be negative definite. In this general case, the PSU and SCH 
correlation coefficients 




p^,p(PSU) = {q^Ip(PSU)(I-Rp(PSU)]q + q^IRp(SCH)q} r {q^Zp(TOT)q} 



= [6cp(PSU) - ^cp(S^")l 

and 

• Pcp(S€H) = {q%(SCH)[i-Rp(SCHm + q^IRp(STU)q} - {q^Zp(TOT)q} 

= [6^p(SCH) - Ccp(STU)] . 
may be negative if the combined stratification and without replacement or 
PMR selection effects t,^^ f^oro the subsequent stage swamp the clustering 
effects 6^Q. The general case also has the primary stage stratification 
and without replacement or PMR selection effect 

Pcp(O) = {q^ IRp(PSU) q}/{q%(TOT)q] 

which is expected to be negative and which has a large coefficient (nsm-1) 
in the design effect expression. 

The empirical results in chapter 3 of this report show that for simple 
constrasts among NAEP P-values and for weighted-least squares coefficients, 
a substantial fraction of the design effects are less than 1. Lepkowski 
and Landis also observed numerous contrast DEFFs less than 1. In fact, the 
OMNI data had mean contrast DEFFs for the ten dependent variables they 
explored' ranging from 0 . 75 to 1 .19 with an average of 0.99. In such 
instances, the SRS based chi-squared statistics are smaller and less sig- 
nificant than -the design based chi-square. While one might attribute some 
of these DEFF values less than 1 to negative bias in the design based 
Taylor Series variance approximation, we feel that the incidence of such 
cases is too great to be totally explained in this fashion. Furthermore, 
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the variance formula used with the Taylor Series linearization to produce / 
the estiroatfes of Vp(DES) use d in this report would overestimate the va ri- 



ance of a linear statistic since it assumes that primary units were selec- 
ted* two or three per stratum with replacement, when in fact they were 
selected without replacement. Specifically, the linearized single draw 
variates are formed separately by primary stratum h such that 

z^^(hijk) = M(hij) X^,(hij) [Y^Xhij) - P(dr).]/X(d) (t)(hij) 
where (j)(hij) = (})(hi) (j)(j|hi) denotes the nonresponse adjusted single draw 
probability for sample school j of sample PSU(i) based on n(h) = 2 or 3 PSU 
selections from primary stratum h and c(hi) school selections from sample 
PSU(hi). The PSU level averages 

0 

c(hi) m(hij) 
z(hi..) =1 I z(hijk)/c(hi) m(hi.i) 
j=l k=l 

are then formed with m(hij) denoting the number of package respondents from 
sample school (hij). The P-value covariance matrix is then estimated by 
the between PSU within stratum mean square 



^ H n(h) /V /N ^ 

Vp(DES) =11 [z(hi..)-z(h...)] [z(hi..)-z(h...)]Vn(h)[n(h)-l] 
h=l i=l 

H 

= I MS (PSU|h)/n(h) 
h=l 



where 



n(h) /V 

z(h...) = I z(hi..)/n(h) 
1=1 



is the primary stratutn-h mean of the linearized single draw variate vector. 
If the within stratum h primary selections had been with replacement, then 



-ERIC 



-46- 52 



the, variance estimator above would be unbiased for a linear statistic like 
fh ^ ly^rtor nf HnnriAin by respon se option t otals Y("*") ♦ Since the NAEP pri- 
maries were selected without replacement, the stratum h effect of without 
replacement selection, namely ZR^(PSU|h), is not accounted for. This 
matrix is expected to be negative definite so that its exclusion f rom:_the::::::: 
variance function will enlarge the variance of any contrast. 

In the Year 13 NAEP primary sample , where sequential PMR selections 
were made from a judicously ordered primary frame, the pseudo-strata formed 
by pairing neighboring selections down the ordered listing should also lead 
to some positive bias in the variance approximation due to ignoring the 
deeper implicit stratification. Jo explore this issue further one could 
contrast Wald statistics based on the Taylor Series covariance matrix 
estimat-or with Wald statistics derived from Balanced Repeated Replication 
(BRR) covariance matrix estimators. Krewski and Rao*s (1979) small sample 
comparisons of TSL and BRR variances for combined ratio estimators suggests . 
that TSL generally has a negative bias while BRR has a positive bias under 
the model 

Y(hi) = a(h) + p(h)X(hi) + e(hi) 

with 

E [e(hi)|X(hi)] = 0 

and ^ 

E [e2(hi)|X(hi)i = 6^X(hi)'' . 

Rao and Krewski show that the absolute bias comparison favors. TLS when 

t S 1. When t = 2, the BRR variance estimators have smaller absolute bias. 

In terms of mean-squared error, the results of Krewski and Rao (1979) and 

Frankel (1971), suggest .that the TSL variances are generally more accurate 

o 

estimators. On the other hand, Frankel (1971) and " Campbell and Meyer 
(1978) show that in reasonably small samples BRR may produce more robust 
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o * ■ . ■ 

J 

inferences in terms of achieving the desired significance level. Direct 






y ■■ ■ ^ . 

TSL and BRR comparisons would shed some light on the TSL negative bias 






• 

potential. Unfortunately such comparisons were beyond the scope of the 






current project. / ^ 






2.4 Testtirg^Bilanced iits Via Dummy Variable Regression 






An alternative mode of analysis for exploring the effect of domain 






classifiers on the Y^(£st) zero-one, correct response indicators has been 






referred to as' "Balanced Fitting" by NAEP analysts. This approach utilizes . 






dummy variable regression models of the form , ^ . . 






Y^(£st) = X(£st) B , 






where the row vector X of independent variables includes a leading 1 for ' 






the intercept parameter and zero-one indicator variables for parameters 






associated with the levels of student *s race, sex, parentis qduca/tion, and 






type of community- where the school is located. The proper ^sample design 






based analysis for testing the significance of such regression coefficients 






has been specified by Folsom (1974) . The universe level least-squares 






solution for the vector of regression coefficients B is specified in terms 






of the universe level' left and right hand sides of the so-called normal 






equations; namely 






T N S(£) M(£s) ^ ^ 
(X X) =11 I X(£st) X(-£st) 
2=1 s=l t=l 






and . * , , • ''"^ 


- 9. 




N S(£) M(£s) . , ■ " 
(X Y) =1.1 I X(£st)^Y(£st) . > . - 

£=1 s=l t=l ~ ' , 






T 

With the regression model parameterized such that (X X) is nonsingular, the 






B vector is defined as 

B = (X^X)"-^(x''^Y) •■ 
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with ( ) ^ denoting matrix inversion. 

In terms' of the balanced three stage analogue of the NAEP design 
explored in the previous sections, the unbiased sample estimators for the 
left and right hand sides are formed from, the following single draw vari- 
ates 



N S(£) M(£s) 

(x'^'x)... = II I a„ . (ijk)M(£s)XUst)'^X(£st)/(t)(As) 
^J*" £=1 s=l t=l '^^^ 



and 



„ N S(£) M(£s) T, 

(xy)... = III a„ . (ijk)M(£s)X(£st)'Y(£st)/(|)(£s) 
"-.^^ £=1 s=l t=l '^^^ 



The corresponding unbiased estimators are formed as the sample means 



rr. n c m ^ 

(x x)._ = I I I (x x) /ncm 



i=l j=l k=l 
and 

n c m _ 
(xV)... = 111 (XT)... /ncm . 
■ i=l j = l k=l . ^•^'^ 

The associated sample estimator for the vector of regression coefficients is 

B = (x^x)"-^ (x^y) 

To 'approximate the sampling variance of B, the following Taylor .^series 
linearized variate was derived independently by Folsom (1974) and Fuller^ 
(1974): . 

/ 2(£st) = (X^X)"-^ X(£st)^ e(£st) 

where 

e(£st) = [Y(£st) - X(£st)B] 
denotes the prediction error or deviation from regression for student list 
unit (£st) . The corresponding list unit single draw variate vector is 
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qUst) = Mas)9(Ast)/(t)(£s) . 

Substituting these linearized single draw variate vectors for the z(£st) 
vectors used previously to define the Ip(STAGE) covariance and ZRp(STAGE) v 
cross-covari'ance component matrices for the vector P of estimated domain 
P-V3lues, one obtains an analogous set of Ig(STAGE) covariance and ZRg(STAGE) 
cross-variance components . Recalling the general form for the total co- 
variance matrix I-.(TOT), one can^show that 

D 

N S(£) M(£s) 

I^(TOT) = M(++)(X^X)"'^ {1 1 C(^s) 1 X(£st)^XUst)e(£st)2} (X^X)^-^ . 

jz=i s=i., . t=i r 

where 

t,Usy= {M(£s)/M(++)] -7 (|)(£s) . • 

This result derives from the fact that 

N S(£) M(£s) . 
' ' = I Z (t)(J^s) f q(£st)/M(£s) = <l> , 

- *** £=1 s=l t=l 

with <P denoting the null vector. For a self -weighting sample with = 1 

for all (£s), the total covariance component matrix 1^(101) is equivalent 

to the simple random sampling covariance matrix 

N S(jZ) M(£s) 

I CSRS) = M(++)(X^X)''^{ 11 1 X(£st)^XUst)e(iist)2] (X^X)""^ . 

£=1 s=l t=l ^ 

The estimated linearized variate for B is defined as follows 
q(£st) = MUs) (x^x)^5_X(ilst)^r(£st)/(t)(£s) 

with 

r(£st) = [Y(£st) - X(£st) B] 
denoting the. observed sample residuals. Composite component matrices 
Sg(PSU), Sp(SCH), and Sg(STU) are estimated from the analogous ANOVA type 
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matrix , mean squares MSg(PSU), MSg(SCH), and MSgCSTU). These B mean-square 
matrices are defined by analogy with the corresponding MSp matrices using 

N S(£) M(£s) 
q(ijk) =11 1 a . (ijk)qUst) 
£=1 s=l t=l 

in place of the 2(ijk) linearized single draw variate vectors. 

A consistent estimator for Zg(TOT) is obtained by recasting Zg(TOT) in 

a form involving the sample weights W(£st) = M(£s)/ncm (|)(£s); that is, 

N S(£) M(£s) J 
Z (TOT) = ncm { Z 1 1 W(£st)q(£st)q(£st) } 
^ £=1 s=l t=l 

N S(£) M(£S) rr. 

= fwM2(++) {11 1 W(£st)2(£st)2(£st) /W(+++)} 
£=1 s=l -1=1 

where 

N S(£) M(£s) 
W(+++) =11 1 W(£st) 
£=1 s=l t=l 

is the universe weight sum; 

r ui = W(+++)/M(++) 

is the universe level average weight, and 

f ,= ncm/M(++) 

is the overall sampling fraction. As before, the unequal weighting design 

0 

effect is estimated by ^ 

ncm ncm «" 

fw = ncm 111 W(ijk)2/[ 111 W(ijk)]2 . 
i=l j=i k=l i=l j=l k=l 

TRfe matrix inside of curley brackets, say SWg(TOT) is estimated consistently 

by^ 



\^ 'ncm ^ .^ „ncm. 

\SW-(TOT) = 1 11 W(ijk)22(ijk)2(ijk)V 111 W(ijk)2 
^ • i=l j=l k=l . i=l j=l k=l 
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where 



Q(ijk) = ix^x)']^ X^(ijk)r(ijk) 



is the estimated Taylor Series linearized variate without the division by 
our single draw probability (|)(£st) = (|)(£s)/M(£s) . 

The simple random sampling covariance matrix Ig(SRS)/M(++)^ is simi- 
larly approximated by 



nc m ^ - ^uc.m 

S (SRS) =111 W(ijk)Q(ijk)Q(ijk)V III W(ijk) . 
i=l j=i k=l - - i=l j=l k=l 

Notice that SWg(TOT) is the weighted sample mean of the 2(ijk)g(ijk) 

matrices using squared weights W(ijk)^ while S-,(SRS) is the comparable 

weighted average based on the original sample weights. For the statistic 

B, the effect of unequal weighting 

= Ijj(SRS)"^ Ig(TOT) 



is estimated by 

; Wg = fwSg(SRS)"^ SWg(TOT) . 

The generalized design effect matrix for B has another component that 
arises from the typical model based least-squares analysis. Assuming that 

[W(ijk)]^(ijk) = [W(ijk)]^ X(ijk)B + e(ijk) 
with errors having zero expectation and common variance conditional on 
the given set of X(ijk) and W(ijk) variables, ordinary least'-squares theory 
produces our weighted B coefficients, and the model based covariance matrix 

V^(MOD) = (x'^x)"''^ 52 
B ^ • • • e 

where 
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n c m 

a2 = I I I W(ijk)r(i-.jk)2/(ncm - p) 
^ i=l j=l k=l " 

is the residual mean square of the W transformed variables. Recognizing 
that 

asm V-(SRS). = M(++) (x''^x)"-^ { I,^W(ijk)x''^(ijk)X(ijk)r(ijk)2} (x'''x)"^. 
■ ^ ijk 

with the total student population size M(++) estimated by 

M(++) = I W(ijk), 

one can write the estimated model effect as « 



Mg = Vg(MOD)"^VgCSRS) 



T v-1 



= [■(ncm-p)/ncm]{ I W(ijk)x' (ijk)X(ijk)r (ijk)2/ou)2} (x x) 



where 



ou)2 = I W(ijk)r(ijk)2/ I W(ijk) 
^ ijk ijk 

is the weighted residual mean square. Under the model where E{r(ijk)2} = (j 
for large samples, the model expectation of Mg given the sample will be 
approximately 1. 

For a set of linear contrasts among the B coefficients, say CB, one 
can define corresponding component matrices 

S^,g (STAGE) = C Sg( STAGE )C^ 
I^,g(TOT) = C Ig(TOT)C^ 
I^,g(SRS) = C Ig(SRS)C^ 

and 

V^,g(MOD) = C Vg(MOD)C^ . . 
These components lead to an estimated generalized design effect matrix of 
the form 
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DEFF(CB) = M^^iL^^[I+(ncm-l)R^^(0)+(cm-l)R^^(PSU)+(m-l)R^^(SCH)] 

where 



and 



Rcb(O) = [I-RcB(PSU)-R^g(SCH)-R^g(STU)] 



R^g(STAGE) = I^g(TOT)"^Sgg(STAGE) . 



With these results, one can again write the asymptotic distribution of the 
ordinary least squares model based test statistic as 



xJqj^(CB) - 1 M^g(a)i^g(a)[l+(ncm-l)p^g(0)+(cm-l)p^g(PSU)+(m-l)p^g(SCH)]x^ 
a=l 

where the are independent single degree of freedom central chi-square 
a 

variables and the coefficient components are of the form 

where £ is the a-th row of the left hand eigenvectors of DEFF(CB) and r 
^ '^a ^ '^a 

is the a-th column of the corresponding right-hand eigenvectors. 

The empirical results presented in chapter 3 for NAEP balanced fit 

V 

parameters suggests that the design effects for these statistics are gen- 
erally greater than one. We suspect that the extra model effect components 
M^g(a) contribute substantially to this result, 
2 .5 Inference for NAEP Package Means 

In order to increase the precision of subgroup comparisons , NAEP 
analysts have turned to averages of single exercise P-values . Averaging 
across exercises (e) within packages (indexed by u) should reduce the 
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variance inflating effect of stochastic response errors. Averaging exer- 
cises across packages has' the potential to substantially reduce sampling 

- ' o 

errors, since each distinct package contributes a nonoverlapping sample of 
approximately 2,600 students. Recalling the definition of an estimated 
NAEP P-value presented in section 2.1, a within package average across 
-exercises labeled e = 1, 2, . . . E(u) can be written in terms of the 
weighted mean of a. student level proportion correct variable. With Y^^(hijk) 
denoting the correct incorrect response indicator for exercise e of package 
u from sample school j of PSU(i) as administered to sample student (hijk) 
in primary stratum (h) , the weighted package u mean for domain d is 

E(u)vs , 
P^.(d) = I P^^(d)/E(u) 
6=1" 

H n(h) c (hi) m (hij) E(u) ' . ^ 

={11 "I W^(hijk)X^^(hijk)[ I Y^^(hijk)/E(U)]}/X^(d) 

h=i i=i j=l ^ k=l " . e=l 

where m (hii) denotes the number of package u respondents from the j-th 
u 

cooperating package u school from PSU(hi) with j ranging over c^(hi) such 

schools. The denominator of P^^Cd) is the package u weight sum for domain d 

o 

members . 

T 

To estimate the design based covariance matrix for the vector P(u) = 
[P^.(l), ' . . , P^.(d), • ' • J ^u-**^^^ package u means, the student 

level P-values 

oE(u) 

Y (hijk) = 1 Y^^(hijk)/E(u) 
e=l 

are used to form lineari2;ed variates 
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^^^(hijk) = M^(hij)X^^(hijk)[Y^ (hijk)-P^ (d)]/X^(d)(t)^(hij) 

= n(h)c^(hi)m^(hij)W^(hij)X^^(hijk)[Y^.(hijk)-P^.(d)]/X^(d) 

These linearized variates lead to the following Taylor Series covariance 
matrix estimator based on the paired ^ith replacement PSU selection model 



H - 
vg(DES) = I MSg (h)/n(h) 
h=l ^ 

where ' 

n(h) 

MS^(h) = I [p (hi..)-p„(h...)][p„(hi..)-p„(h...)] /[n(h)-l] 

1=1 



is the primary stratujn h contribution to the covariance matrix. To produce 
the PSU(hi) level mean vectors Q (hi ), the student level vectors of D 

^U • • ' tf 

linearized variates 



y^(hijk) [p^^(hijk), . . . , p^^(hijk), . , pp^(hijk)] , 

are first averaged over the m^(hij) students responding to package u in 
cooperating school (hij) and then these 'school level mean vectors are 
averaged over the c^(hi) cooperating schools from PSU(hi) phat are assigned 
package u. 

To allow., for the consideration of item P-value averages extending 
across packages^, say 

U - 
P_(d) = 1 P^.(d)/U , ^ 
u=l 

the full covariance matrix for the extended vector 

P' = [P(l)\ . . . , P(u)\ . . . , P(U)] 
of package level domain means is required. This extended covariance matrix 
can be produced simply by extending the PSU level linearized vector means 
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to include subvectors for each package u involved in the average;, that is, 
one defines 

U(hi = [p. (hi )'^, . . . , M,7(hi ), . . Pu'Chi )'^] 

and forms Vp(DES)by substituting U(hi_) and U(h_ J for the associated 
package specific vectors in the definition of V-(DES). Note that the 
vector of D cross-package means is a simple linear transformation of the P 
vector of the form 

P = CP 

with the d-th row of C having the form 

CCd) = [6^(d), . . . , 6^(d), . . . , 6^(d)]/U 
where 6 (d) is a (1 x D) row vector with a 1 in position d and zeros else- 
where. The estimated design based covariance matrix for P is therefore 

V^(DES) = CVp(DES):C^ . 

The simple random sampling covariance matrix V^(SRS) for P is 
diagonal with d-th diagonal element 

V^(SRS) = { 1 Sj(d)/m^(d)}/U2 
u=l 

^ where s' 

H n(h) c (hi) m (hi j ) 
S2(d) =1 1 ""l W (hijk)X (hijk)[Y (hijk)«P (d)]2/X^(d) 

h=l i=l j=l k=l \ - 

is the estimated subpopulation d variance of the Y^^(hijk) student^ level 
proportions correct. Recall that X^(d) is the package u estimate of the 
universe count of students in subpopulation d. 
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The chi-square adjustment factors proposed by Rao and Scott based on 
the average eigenvalue of the generalized design effect matrix 



DEFF[P] = VrCSRS)' VsCDES) 



have the form 

D 

AVED[P] = 1 DEFF[P(d)]/D 
d=l 



were DEFF[P(d)] is the design effect for the d-th element of P. 

In addition to the weighted least squares/Wald statistic type analysis 
directed at the P vector, balanced fit type analyses directed at cross- 
exercise and cross-package means have been pursued. For these analyses, 
the student level P-values Y^^(hijk) for all the U package, samples involved 
in the cross-package average were used as the dependent variables in a pair 
of main effect regression models- For a fully interactive regression model 
including for example race, sex, and parents education, the model based 
predicted values for each race by sex by parents education cell (c) would 
have the form of a weighted combined ratio mean 

0 

U 

Y(c)' = { HzUlill^ J 

I I W (hijk)X (hiik) 
^ u=l hijk 

= { I X (c)P (c)/ I X (c) }. 
u=l • u=l 

where X^(c) is the package u sample estimate of the universe level student 
count for subpopulation c. The main effect balanced fit models yield 
reduced model approximations of the combined ratio means Y(c) . The (DES) 



covariance matrix for the main effect parameters fit to these student level 
P-val es were obtained using the design based regression^ procedures des- 
cribed in section 2.4. The corresponding covariance matrix applicable for 
a standard model based regression analysis were obtained by irunning the 
transformed variables W^(hljk)^^^, (hijk) and W^(hijk)^^(hijk) through an 
ordinary least squares package yielding 

Vp(MOD) = (x^x)^^^^ gI 

where is the residual mean square among the transformed Y variates and 
e 

(x^x)"""^ denotes the inverse of ^the weighted sums of squares and cross 

products matrix forming the left-hand sides of the normal equations. 
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3. EMPIRICAL INVESTIGATION 

3.1 Analysis Items and Subgroups 

Intially, five NAEP exercises per age class were selected for analysis 
from the Year 09 Mathematics Assessment. One item was selected from each 
of the following five content objectives: , 

*> 

A. numbers and numeration, 

B. variables and relationships, 

C. size, shape, and position, 

D . measurement , and ' 

E. other topics . ^ . 

Copies of the selected exercises are included in Appendix A. Each item was 
receded one for correct and zero for incorrect. An additional score was 
defined for each student as the proportion of the items analyzed on a 
package that the student . answered correctly. This score was analyzed 
within each age class to form three mean scores for analysis. 

Four domain or subgroup defining variables were also selected. These 
were, with their corresponding levels:. 

Sex Race 

Male White 
Female Other 

Type of Community (TOC) Parental Education (PARED) 

Extreme Rural Not High School Graduate 

Metro High School Graduate 

Other Post High School 

3 . 2 Analyses 

The ultimate goal of this study was to compare sample design based 
analyses of NAEP data with those assuming a simple random sample. This was 
done separately for two analytic methods. The first analytic method that 
will be discussed is the Wald statistic/weighted least squares approach. 
This will be followed by a discussion of the work done for the balanced 
fits analyses. 
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The Wald statistic/weighted least ^squares approach, described earlier 
in section 2.1, proceeds by first estimating a vector of domain statistics 
and its corresponding cov^riance matrix. Various hypotheses concerning 
this vector can then be evaluated. Two vectors of domain means were formed 
for each of the 15 item scores and the three mean scores. The first vector 
contained 12 elements corresponding to the complete cross-classification of 
Race, Sex and Parents Education (PARED). The second vector was derived 

a. 

from the cross--classif ication of Sex, Type of Community (TOC) and PARED and 
was of length 18. For the 15 item scores, these vectors consisted of 
simple proportion correct p-values. Two covariance matrices were then 
estimated for each vector. One based upon the actual sample design and the 
other assuming a simple random sample of. students. The details of the 
estimation process were provided in Chapter 2. 

At this point several exercises were excluded from the study because 
their estimated covariance matrices were singular". For the Race'^'^SeX'^TARED 
cross-classification only item N0317A was excluded. However, for the 
Sex^^TOC^^PARED cross-classification it was necessary to exclude items 
N0227A, N0317A, N0323A, T0224A, and S0121A. 

A linear model was then fitted, via weighted least squares, to each of 
the remaining domain mean vectors. For the Race-Sex-PARED domain cross- 
classification vectors the model contained the main effects of Race and 
Sex, a linear effect of PARED and the four possible two- and three-way 
interactions among these three effects. The Sex-TOC-PARED domain classifi- 
cation model had the same form except that TOC was substituted for Race. 
These models were fitted two ways — one weighted with the design based 
covariance matrix and the other weighted with the simple random sampling 
covariance matrix. The lack of fit of each model and the significance of 
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each effect in the model was then assessed. These tests are labelled one 
through ieight in Tables 3-1 and 3-2. 

In addition, nine other hypotheses were considered and are labeled 
nine thr^ough 17 in Tables 3-1 and 3-2, These hypotheses were tested via 

direct contrasts of the domain means. The tests labeled "average'^ (numbers 

1 

10, 11, il2 and 13) average the effect over the combined levels of the other 
two variables. On cthe other hand, the "nested" tests (numbers 14, 15, 16 
and 17) )^est for all the indicated simple effects being simultaneously null 
over the combined levels of the other two variables. 

Threel test statistics were entertained for each hypothesis.. The first 
test was a^ Wald statistic chi-squared based upon the actual NAEP sample 
design. A second Wald statistic chi-squared was also calculated assuming a 
simple randbm sample o,f students. Finally, the simple random sampling 

chi-squared Us adjusted as shown in section 2.2 by dividing by the average 

1 

design effecjt to obtain the third test statistics. These three test 
statistics wire calculated for each hypothesis for 14 NAEP items and three 
mean scores ior the Race'>Sex-PARED cross-classification, as well as for 10 
NAEP items pljus three mean scores for the Sex-TOC-PARED cross-classification 
All of these test statistics are shown in Appendix B along with their 
asspciated significance levels assuming that each has a chi-squared distri- t 

i 

bution. Thej test numbers in Appendix B correspond to those in Tables 3-1 

f 

and 3-2. I ^ ^ . , 

' Turning now to the balanced effects^^ analyses , the 15 NAEP items plus 
three age delated mekn scores discussed earlier were studied. As noted in 
Chapter 2, jthe balanced effects methodology is used in a regression setting . 
to ass-ess /the significance of a particular effect after, adjusting for the 
other factors in the model. For this portion of the study',, each of the 
NAEP iten^^ scores and three mean scores were- regressed on two models. One 



Table 3-1. 


Hypothesis Tests 


for the Race-Sek-PARED Cross- 




Class ifica t ion 




i6Su KumDer 


a . I . 


Description 


Linear Model Tests 




"* ' ' — ^ 


1 


4 


Lack of fit 




i 


Race 


•a 


.1 


oex 


4 : 




r^AivCiU linear 




1 


* ^ x\ace'*oex . 


o 


1 


Kace^^rAKHiU linear 


/ 


1 ^ 


Sex'* PARED linear 


o 
o 


1 <» 


Kace^Sex'^PARLD linear 


Contrast Tests 






q 


11 


^ Hiiceiis et^uai 


10 


1 


Average Race effect 


11 


1 


/ ^ Average Sex effect 


12 


'2 


/ Average PARED effect 


13 


1 


/ Average PARED linear effect 


14 • 


6 


/ Nested Race effect 


15 


6 


/ Nested Sex effect 


16 


8 


/ Nested PARED effedt 


17 


4 


^ Nested PARED linear effect 



/:■: . 

\- ■ ■ 

■ / 

/ 
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Table 3-2. Hypothesis Tests for the Sex-TOC-^PARED Cross- 
Classification 



Test Number d.f. Description 



Linear Model Tests 

1 '6 Lack of fit 

2 1 Race 

3 2 TOC 

4 1 PARED linear 

5 - 2 Sex^TOC 

6 ' 1 Sex"PARED linear 

7 2 TOC-PARED linear 

8 ^2 Sex*TOC" PARED linear 

Contrast Tests ^ 

9 17 All cells equal 

10 1 Average Sex effect 

11 2 Average TOC effect 

12 2 Average PARED effect 

13 1 Average PARED linear effect 

14 9 Nested Sex effect 

15 12 Nested TOC effect 

16 12 Nested PARED effect 

17 6 Nested PARED linear effect 



model contained the main effect of Sex, Race and PARED, while the other 
contained the main. effect of Sex, TOC and PARED. The three partial F-tests 
for each effect in the model controlling for the other two effects were 
then considered for each model and mean or item score. 

Each model, and hence each F-test, was fitted in three different ways 
for comparison. One approach employed the sampling weights and the Taylor 
series variance estimation technique discussed in section 2.4. This yielded 
strict design based significance tests. Test statistics were also obtained 
using a standard regression package (the GLM procedure of SAS) ignoring 
both the sample design , and the sampling weights. This approach produces 
biased estimates of the regression coefficient, as well as producing infer- 
ential statistics under inappropriate standard regression assumptions. 
Finally, a weighted version of the SAS GLM procedure was used. This process 
properly incorporates the sampling weights to produce the correct statisti- 
cally consistent estimates of the regression coefficients while still 
appealling to inappropriate standard regression assumptions for inference. 
Since the statistical package used for this last portion of the balanced 
effects analysis uses unweighted sample counts to calculate its degrees of 
freedom, the analyses so obtained are equivalent to those that would have 
resulted from first scaling the sampling weights so that they summed to the 
unweighted sample size and then using a statistical package that used the 
sum of the weights as its total degrees of freedom. The balanced effect 
F-tests along with their significance or probability levels are presented 
in Appendix C. 
3.3 Results 

3.3.1 Wald Statistic/Weight Least Squares 

The design effects (DEFFs) for. each domain p-value and mean score used 
in the Wald statistics/weighted and least squares analyses are summarized 



in Tables 3-3, 3-4, and^ 3-5. Each table presents the minimuni, median, 
maximum and mean DEFFs for a particular NAEP item or mean score across the 
levels of the indicated domain defining cross-classification (i.e., 
Race-Sex-PAKED or SexnOC^PAKED) . The design effects reported in these 
three tables are consistent with previous NAEP , experience and tend to 
average around 1.4. Also, as discussed in section 2.2, the mean DEFF's 
given in the last column of each table are the exact quantities proposed by 
Rao and Scott (1981) and Fellegi (1980) for adjusting simple random sampling 
(SRS) based Wald Statistics chi-squareds to reflect the effects of the 
sample design. These are the adjustment factors used in the subsequent 
discussion. 

As was noted in section 3.2, two different methods of analyses or < 
hypothesis testing often used by researchers was considered within the Wald 
statistic/weight least squares context. The first fitted a linear model to 
the estimated domain statistics. Relevant hypotheses were then tested via 
contrasts of the estimated linear model parameters. The parameters were 
estimated weighting inversely proportional to the SRS covariance matrix of 
the domain statistics to obtain the SRS test statistics. Another set of 
parameter estimates was obtained by weighting by the inverse of the design 
based covariance matrix and the asymptotically correct test statistics were 
calculated. The second method of analysis evaluated hypotheses via direct 
contrasts of the domain statistics. Again this was first accomplished 
using the SRS covariance matrix to obtain the SRS test statistics, and was 
then repeated using the design based covariance matrix to obtain the 
asympotically correct tests. Results in the rest of this sec^tion will be 
presented separately for these two modes of analysis (i.e., contrasts of 
linear model coefficient and contrasts of cell means). 
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Table 3-3. NAEP Item Design Effects for the Race * Sex * PARED 
Cross -Classification 



NAEP 
Item 



Minimum 
DEFF 



Median 
DEFF 



Maximiun 
DEFF 



Mean 
DEFF 



N0222A . 79 

N0227A .80 

N0305C .62 

N0323A .59 

T0105A .91 

TOllOA .56 

T0203A .99 

T0223A .69 

T0224A 1.00 

S0108A .63 

S0117A ■ .61 

S0121A .39 
S0206A - .72 

S0225A ^ 

Average .71 



1 .23 
1.36 
1.39 
1.27 
1.50 
1.26 
1.72 
1.13 
1.31 

.94 
1. 17 
1.09 
1.25 

.84 
1.25 



3.08 
1.94 
1.93 
1.67 
2.84 
2.38 
2.29 
2.32 
2.82 
1.99 
2.44 
3.71 
3.44 
1.83 
2.48 



1.48 
1.40 
1.35 
1.14 
1.63 
1.43 
1.66 
1.28 
1.47 
1.11 
1.23 
1.37 
1.40 
.99 
1.35 



ERIC 
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Table 3-A. NAEP Item Design Effects for the Sex - TOC - PARED 
Cross -Classification 



NAEP 


Minimum 


Median 


Maximum 


J 


Mean 


X t em 


DEFF 


DEFF 


DEFF 




DEFF 


N0222A 


• 21 


1. 


17 


2. 


49 




1.25 


N0305C 


.37 


1. 


53 


2. 


21 


■ 


1.35 


T0105A 


.49 


1 


40 


4. 


32 




1.61 


TOllOA 


.64 


1- 


28 


3 


02 




1.31 


T0203A 


.27 


1 


36 


4 


46 




1.62 


T0223A 


.68 


1 


14 


2 


10 




1 

i . ZD 


S0108A 


.44 


1 


,03 


2 


01 




1 . 14 


S0117A 


.35 


1 


11 


2 


.14 




1.14 


S0206A 


.48 


1 


.53 


4 


17 




1.66 


S0225A 


.47 




.93 


2 


.37 




1.04 


Average 


.44 


1 


.25 


2 


.93 




1.34 



Table 3-5. Mean Scores Design Effects 





Minimiun 


Median 


Maximiun 


Mean 


Model/Age 


DEFF 


DEFF 


DEFF 


DEFF 


RACE*=^^SEX"PARED 










9-year-olds 


.57 


1.45 


3.32 . 


1.50 


13-year-olds 


.78 


1.31 


2.33 


1.46 


17-year-olds 




1.09 


2.. 5 7 


1.16 


Average 


.61 


1.28 


2.74 


1.37 


SEX-TOC-^PARED 










9-year-olds 


.80 


1.52 


3.47 


1.66 


13-year-olds 


.59 


1.50 


3.57 


1.66 


17-year-olds 


.75 


1.30 


2.61 


1.45 


Average 


.71 


1.44 


3.32 


1.59 



0* 
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For each hypothesis test entertained in this •portion of the investiga- 
tion, the ratio of the SRS based Wald statistic to the asymptotically 
correct sample design based Wald statistic chi-squared was calculated. 
These ratios are another measure of the effect of the sample design and are 
referred to in the remaining tables as hypothesis test design effects. Two 
issues will be addressed by way of these test DEFFs. First, an indication 
of the ordinal relationship between the two test statistics will be sought. 
That is, does the SRS statistic tend to be generally smaller or larger than 
the design based chi-squared? Second, are the test DEFFs fairly ^constant , 
at least within an item or mean score? This second point is important if a 
simple^ multiplicative adjustment to the SRS test statistics is to be 
successful. Tables 3-6, 3-7, 3-8, present a summary of the test DEFFs for 
each mean or item score for the indicated cross-classification. The 
mininiom, median, maximum and mean test design effects are shown separately 
for linear model coefficient contrasts (test numbers 1 through 8. in Tables 
3-1 and 3-2) and cell mean contrasts (test numbers 9 through 17 In Tables 
3-1 and 3-2) . 

The most striking feature of these three tables is the extreme 
instability of the test DEFFs for linear model coefficients. In virtually 
every case the mean is far greater than the median, indicating a skewed 
distribution with a long right-hand tail. It appears that adjusting the 
SRS test statistic for the linear model coefficient contrasts will not 
prove fruitful because of the extreme range they cover. This may result 
from using the SRS covariance matrix to estimate othe linear model parameters 
for the SRS test statistic. This process does not properly account for the 
correlated aature of the domain statistics and leads to less precise 
estimates of the model coefficients. Conversely, Tables 3-6, 3-7, and 3-8 
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Table 3-6. Hypothesis Test Design Effects by NAEP Item for 
the ^Race - Sex - PARED Cross-Classification 



Contrast of Linear 

NAEP Model Coefficients Contrast of Cell Means 

Item Minimum Median Maximum Mean Minimum Median Maximum Mean 



N0222A 


.04 




.82 


5 


.42 


1 


.41 


. 19 




.74 


1 


.81 




.88 


N0227A 


.00 




.57 


900 


.26 


112 


.96 


.23 


1 


.02 


1 


.60 




.88 


N0305C 


.09 




.57 


18 


.69 


3 


.88 


.62. 


1 


.33 


2 


.40 


1 


.38 


N0323A 


.00 




.48 


1 


.08 




.57 


.51 


1 


.08 


2 


.01 


1 


: 12 


T0105A , 


.32 . 




.99 


15 


.73 


4 


.02 


.44 


1 


.16 


1 


.98 


1 


.27 


TOllOA 


.16 




.63 


1 


.72 




.81 


.56 


1 


.18 


2 


. 18 


1 


.19 


T0203A 


.10 




.86 


2 


29 


1 


.03 


.53 


1 


.51 


2 


.21 


1 


.50 


T0223A 


.49 


5 


. 10 


284 


87 


45 


.21 


.72 


1 


.11 


1 


.63 


1 


.10 


•T022AA 


.80 


1 


.68 


34 


09 


9 


.05 


.65 


1 


.10 


2 


.41 


1 


.27 


S0108A 


.03 




.71 


47. 


13 


6 


47 


.55 




84 


1 


50 




93 


S0117A 


.19 




.59 


3 


62 




97 


.53 




75 


1 


75 


1 


00 


S0121A 


.00 




.47 


26. 


19 


3 


91 


.60 




95 


2 


23 


1. 


19 


S0206A . 


.59 


1 


51 


2. 


67 


1 


58 


.59 


•1. 


10 


2. 


09 


1. 


12 


S0225A 


.34 




65 


2. 


33 




87 


.43 




92 


1. 


09 




84 


Average j 


.23 


1 


12 


96. 


15 


13. 


77 


.51 


1 . 


06 


1. 


92 


1. 


12 



77 



Table 3-7. Hypothesis Test Design Effects by NAEP Item for 
the Sex * TOC * PARED Cross-Classification 



Contrast of Linear 



NAEP Model Coefficients Contrast. of Cell Means 



Item 


Minimum 


Median 


Maximum 


Mean 


Minimum 


Median 


Maximum 


Mean 




Aft 


A 9ft 


55 . 


14 


1 1 


51 


. 1 1 


.48 


■' 9 


ft9 


.75 


rju ju jL» 




1 Dft 


1 A A 

190 . 


oy 




Q Q 

yo 


. 19 


.97 


1 
1 


O 1 


.89 




OA 


60 


6 


97 


1 


70 


. 13 


.39 


3 


.23 

o 


.98 


TOllOA 


.37 


.76 


1 


57 




.80 


.19 


.55 


3 


41 


.84 


T0203A 


.14 


.44 


3 


93 




.91 


.27 


1.08 


1 


.84 


.93 


T0223A 


.22 


1.23 


10 


.30 


2 


.40 


.45 


.86 


1 


.13 


.77 


S0108A 


.02 


.14 




.64 




.22 


.10 


.36 


2 


.62 


.73 


S0117A 


.46 


.97 


2 


.80 


1 


.22 


.03 


.36 


2 


.46 


.70 


S0206A 


.11 


.47 


1 


.27 




.54 


. 10 


.64 


1 


.27 


.59 


S0225A 


■ 05 


.75 


2 


.98 




.98 


.23 


.45 


1 


.43 


.60 


Average 


.20 


1.07 


27 


.57 


5 


.03. 


.18 


.61 


2 


.20 


.78 
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Table 3-8. Hyp^'hesis Test Design Effects for Mean Scores 



Model/Age 


Contrast 


of Linear 


Model Coefficients' 


Contrast of 


Cell Means 




Minimum 


Median 


Maximum 


Mean 


Minimum 


Median 


Maximum 


Mean 


Race^^-Sex'^PARED 




















9-year-olds 


.11 


.22 


-3.74 


.85 


.29 


.91 


1 


67 


1.00 1 


13-year-olds 


.09 


1.86 


7064.23 


885.11 


.59 


• 1.19 


9 




1.26 


17-year-olds 


.00 


.43 


1 1 
1 . 1 D 


. DO 




1 . UQ 


1. 


32 


.89 ] 


Average 


.07 


' ^84 


.2356.38 


295.51 


.43 


1.06 


1. 


74 


1 . 05 


S ex -TOC- PARED 




















9-year-olds 


.23 


.39 


1.39 


.55 


. 19 


.62 


2. 


53 


.91 1 


13-year-olds 


.05 


.50 


' 1.96 


.74 


.17 


.72 


2. 


87 


1.09 


17-year-olds 


\02 


" .65 


223.55 


28.54 


.03 


■ 50 


1. 


27 


.53 j 


Average 


.10 


.51 


75.63 


9.94 


.13 


.61 


2. 


22 


.84 



70 
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indicate that the cell mean contrast hypothesis test design effects tend to 
be more symetricaLly distributed over a narrower range than their linear 
model counterparts. However, they still exhibit enough variation on both 
sides of unity to make a simple multipicative adjustment questionable. 

As indicated earlier, theoretical considerations suggest that the mean 
design effects presented in Tables 3-3, 3-4 and 3-5 may provide serviceable 
adjustments to the SRS test statistics. Thrs conclusion is drawn into 
questron by comparing the standard mean DEFFs in these three tables with 
the average test DEFFs for cell mean contrasts in Tables 3-6, 3-7, and 3-8. 
Almost without exception the mean test DEFFs are less than their correspond- 
ing p-value DEFF average. In addition, • the mean hypothesis test DEFFs are 
generally near unity or less while the standard mean DEFFs are generally 
much greater than unity. This implies that dividing the SRS test statistic 
by the mean design effect will produce a test that is generally much too 
conservative. In fact, the adjustment suggested by Rao and Scott (1981) or 
Fellegi (1980) is in the wrong direction for the examples presented here. 

The hypothesis test design effects are further summarized in Table 3-9 
through^3-12. These four tables display the distribution of the test DEFFs 
over NAEP Items or mean score for each of the hypothesis tests shown in 
Tables 3-1 and 3-2. As was noted before, the linear model tests are very 
unstable. An interesting observation for the cell mean contrast test DEFFs 
is the distinct relationship between the number of degrees of freedom 
(d.f.) for the test and mean test DEFF. The larger d.f. tests have the 
smaller mean test DEFFs. The relationship is almost deterministic. The 
minimum, median and maximum test DEFFs also follow^this distinct relation- 
ship. This observation is surprising in light of the eigenvalue inequality 
presented in section 2.2. This inequality indicates that as the number of 
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Table 3-9. Hypothesis Test Design Effects for NAEP Items by Test Number 
for the Race ''^ Sex PARED Cross-Classification 



Test 



Numbers a 


d. f . 


Minimum 


Median 


Maximiun 


Mean 


Linear node! lests 














1 




. 72 


. 95 


2 


. 23 


1 . 09 






. 31 


. 97 


26 


. 19 


2 . 88 






T / 


. 59 


900 


26 


86 . 14 






. DO 


1.17 


18 


.69 


2.41 


5 




.01 


.58 


1 "7 


1 Q 
JO 


2.59 


6 




.00 


.64 


12 


10 


2.20 


7 




. uu 


A A 


55 


51 


1 U . 1 D 


8 




.04 


.56 


15 


73 


2.67 


Average 




.24 


.74 


131 


01 


13.77 


Contrast Tests 














9 


11 


.19 


.64 


1 


18 


.72 




1 


1.07 


1.62 


2 


41 


.1.71 


11 


1 


.82 


1.25 


2. 


06 


1.38 


12 


2 


.74 


1 . 15 


2. 


40 


1.34 


13 


1 


.51 


1.10 


2, 


09 


1.22 


14 


6 


.44 


1.10 


1. 


94 


1.07 


15 


6 


.43 


.70 


"1. 


12 


.74 


16 


8 


.37 


.72 


1. 


64 


.80 


17 


4 


.72 


.95 


2. 


23 


1.09 


Average 




.59 


1.03 


1. 


90 


1.12 



8i 

-75- 



/ 

/ 

/ 

/ 

/ 

f 



Table 3-10. Hypothesis Test Dei;iga Effects for NAEP Items by Test Number 
for the Sex « TOC* PARED Cross-Classification 



Test 



Numbers 


d.f. 


Minimum 

0 


Median 


Maximum 


Mean 


Linear Model Tests 




c 










1 


6 


.17 


.52 


1- 


53 


- .78 


2 


1 


.09 


.97 


19P 


09 


20.03 


3 


2 


. 11 


.66 


55 


14 


6,30^ 


4 


1 


.42 


.77 


3 


85 


1.11 


5 


2 


.02 


.58 


12 


47 


2,65 


6 


1 


.04 


.80 


45 


56 


6.48 


,7 


2 


.05 


. 69 


4 


42 


1 . 00 


8 


2 


.03 




- 11 


62 


1.87 


Average 




.12 


.70 


40 


59 


5.03 


Contrast Tests 














9 


17 


.03 


.16 




50 


.19 


10 


1 


.35 


.77 


2 


.82 


.98 


11 


2 


.32 


1.31 


■ 3 


23"" 


1.53 


12 


2 


.57 


.85 


1 


38 


.92 


13 


1 


.41 


1.05 


3 


41 


1.34 


14 


9 


.13 


.36 


1 


.08 


.45 


15 


12 


.11. 


.24 




.45 


.27 


16 


12 


.14 


.54 


1 


.13 


.55 


17 


6 


.17 


.52 


1 


.53 


.78 


Average 




.25 


.64 


1 


.73 


.78 
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Table 3-11. Hypothesis lest Design Effects for Mean Scores by Test Number 
for the Race * Sex " PARED Cross-Classification 



Test 
Numbers 



d.f. 



Minimum 



Median 



Maximum 



Mean 



Linear Model Tests 



1 

2 

3 

A 

5, 

6 

7 

8 

Average 

Contrast Tests 

9 

10 

11 

12 

13 
, 14 

15 
-16 

17 ■ ■ 
Average 



11 
1 
1 
2 
1 
6 
6 
8 
4 





.29 


1 


.15 


1 


.83 


1 


.09 




.41 




.44 


3 


74 


1 


.53 




.09 




.15 


. 1 


.16 




.47 


1 


.01 


1 


.06 


1 


90 . 


1 


.32 




.00 




.21 


3 


46 


1 


.22 




.22 




.42 


7064 


23 


2354 


.96 




.01 




.11 


7 


91 


2 


.68 




.20 




.28 


1 


88 




.79 




.28 




.48 


885 


76 


295 


.51 




.42 




.76 




8i 




66 


1 


.23 


1 


.67 


2 


23 


1 


.71 




.79 


1 


.19 


1 


36- 


1 


11 


1 


.08 


1 


.49 


1 


71 


,1 


43 




■ 91 


1' 


.08 


1 


37 


1 


.12 




.97 


1 


.13 


1. 


32 


1 


14 




.49' 




.59 




76 




61 




.40 




67 




76 ■ 




61 




29 


1 


.15 


1: 


83 


t, 'l- 


0,9 




73 


1 


.08 


1. 


35 




05 



Only three observations. 
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Table 3-12. Hypothesis Test Design Effects for Mean Scores by Test Number 
for the Sex - TOC * PARED Cross-Classification 



Test . 
















Numbers 


d..f . 


Minimum 


Median 


Maximum 


Mean 


Linear Model Tests 
















1 


6 






.64 




69 


. bZ 


2 


1 


. 02 




.05 




26 


. 1 1 


3 


2 






.61 




82 


. JO 


A 


1 


1 C\ 




.47 


1 


39 


. Do 


5 


2 


n 

O O 

• 23 




.58 


1 


?6 


. yz 


0 


1 


. ZD 


1 


.32 


223 


55 




7 


2 


.16 




.53 


1 


.24 


.64 


8 


2 


.26 


1 


.11 


1 


44 


.94 


Average « 




.25 




.66 


30 


17 


9.94 


Contrast Tests 
















9 


17 


.03 




.17 




A3 


.21 




1 


.50 




.72 


1 


17 


.80 


11 


2, ■ 


1.27 


2 


.53 


2 


74 


2.18 


12 


2 


.66 




.85 . 


1 


35 


, .95 


13 


1 


.62 




.86 


2 


87 


1.45 


14 


9 


.34 




.61 




89 


.61 


15 


12 


.19 




.21 




51 


.30 


16 


12 


.22 




.48 




60 


.43. 


17 


6 


.54 




.64 




69 


.62 


Average 




.49 

. if 




.79 


l' 


25 


.84 



Oaly three observations- 
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contrasts simultaneously tested (i.e., degrees of freedom) increase the 
mean test DEFF should approach the mean design effect if the adjustment to 
the SRS test is effective. However, the exact opposite relationship is 
observed. As the d.f. increase the mean test DEFF tends to depart further 
from the mean DEFF. This casts further doubt on the appropriateness of the 
mean DEFF adjustment. < ^ 

The Wald statistic/weighted least squares data was also analyzed by 
considering the tables in Appendix D. This appendix presents contingency 
tables of the number of tests which were either accepted or rejected at the 
five percent significance level by the sample design b^secf test versus 
either the SRS test or the adjusted SRS test. Recall that the adjusted 
test was ob'taineil by dividing the SRS test statistic by the appropriate 
mean design effect given in Tables 3-3, 3-4 or 3-5. All three test 
statistics were compared against the chi-squared distribution with the 
appropriate^ degrees of freedom. Appendix D was further summarized by 
calculating the four iditional percents of reaching an opposite conclu- 
sion for each contingency table which are reported in Tables 3"13 through 
3-l6. The last column of the table for cell mean contrasts of NAEP items 
(Table 3-13) indicates that the SRS tests are actually too conservative. 
This seems to be especially apparent for the Sex-TOC-PARED cross-classif ica- 
tion. Approximately 15 percent of. the Race'^Sex- PARED and 32 percent of the 
Sex-TOC^^PARED hypotheses accepted by the SRS test should have been rejected. 
Conversely, approximately ten percent of the hypotheses accepted ^by the 
asymptotically correct sample design based test were rejected by the SRS 
test. This implies that while the SRS test tends to be overly cons^ervative , 
it does not follow that any hypothesis rej ected by the SRS test would be 
rejected by the sample design based test. In addition, nofe that the 
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Table 3-13. Conditional Percent of Contrast Design Based (DB) Tests Versus Alternative 
Tests (AT) Reaching an Opposite Conclusion for NAEP Items 



Cross- Alternate Rejected by AT Accepted by DB Accepted by AT Rejected by 

Classification Test given accepted given rejected given rejected DB given 

, by DB by AT by DB accepted by AT 



x\ace'*oex" r/\KHJ 




















CDC 


















9-year-olds 


10 


.0 


13.3 


18 


8 


14.3 






13-year-olds 


7 


.1 


3.4 


9 


7 


18.8 






1 / ycdr oius 


D 


o 
J 


J . D 


U 


g 


1 1 ft 






ages 


O 
O 


> U 




1 n 

i u 




1 4 A 






rtUJ Ua Lc U c> 


















9-year-olds 


5 


0 


7.1 


18 


8 


13.6 






1 ^ o Ck r r\ 1 He 
J. ^ ycdir UXlia 


7 


1 

t 1 


J . u 


1 7 


Q 
y 


23 S 






17-year-olds 


0 


.0 


0.0 


10 


3 


15.8 






All ages. 


4 


.0 


2.9 


13 


2 


17.2 




Sex-TOC'VPARED 




















SRS 


















9-year-olds 


0 


0 


0.0 


38 


5 


50.0 






13-year-olds 


18 


2 


8.0 


8 


0 


18.2 






17-year-olds 


10 


0 


4.3 


15 


,4 


30.8 






All ages 


11 


5 


5.4 


17 


2 


32.4 






Adj usted 














• 




9-year-olds 


0 


0 


0.0 


61 


5 


61.5 




o 


13-year-olds 


0 


0 


0.0 


28 


0 


38.9 






17-year-olds 


10 


0 


4.8 


23 


1 


40.0 






All ages.^ 


3 


8 


2 . 3 


32 


8 


45.7 





ERIC 
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Table 3-14. Conditional Percent of Contrast Design Based (DB) Test Versus 

Alternative Tests (AT) Reaching an Opposite Conclusion for Mean Scores 



Cross- Alternate Rejected by AT Accepted by DB Accepted by AT Rejected by 

Clcassif ication Test given accepted given rejected given rejected DB given 

by DB by AT by DB ^ accepted by AT 



Race-'^-Sex-PARED 



I) 

00 



Sex'^TOC'VPARED 



SRS 
9-year-olds 
13-year^olds 
17-year-olds 
All^ages 

Adj usted 

9-year-olds 
13-year-olds 
1 7 -year- olds 
All ages 



SRS 

9-year-olds 
13-year-olds 
17-year^olds 
All ages 



0.0, 
0.0 
0.0 
0.0 



0.0 
0.0 
0.0 
0.0 



0.0 
33.3 

0.0 
12.5 



Adj usted 

9-year-olds 0.0 

13-year-olds 33.3 

17-year-olds 0.0 

All ages 12.5 



0.0 
0.0 
0.0 
0.0 



0.0 
0.0 
0.0 
0.0 



0.0 
0.0 
0.0 
0.0 



0.0 
0.0 
0.0 
0.0 



14.3 
0.0 
0.0 
5.0 



33.3 
0.0 
0.0 

12.5 



0.0 

14.3 
0.0 
5.3 



14.3 
0.0 
0.0 
5.3 



33.3 
0.0 
0.0 

12.5 



0.0 
25.0 
0.0 
8.3 



42.9 
50.0 
33.3 
42.1 



60.0 
60.0 
40.0 
53.3 



Table 3-15. Conditional Percent of Linear Model Design Based (DB) Tests Versus Alternative Tests (AT) 
Reaching an Opposite Conclusion for NAEP Items 



Cross- Alternate Rejected by AT ^ Accepted by DB ^Accepted by AT Rejected by 

Classification Test given accepted given rejected given rejected DB given 

by DB by AT by DB accepted by AT 



Race--Sex--PARED 



Sexn0C''^PARED 



SRS 

9-year-olds 3.8 16.7 16.7 3.8 

13-year-olds 16.7 23.5 18. ff 13.0 

17-year-olds 3.4 . 10.0 18.2 6.7 

All ages 7.6 18.2 18.2 7.6 

Adjusted 

9-year-olds 0.0 0.0 16.7 3.7 

13-year-ords 12.5 21.4 31.3 19.2 

17-year-olds 3.4 10.0 , 18.2 6.7 

All ages ^ 5.1 13.8 24.2 9.6 

SRS 

9-year-olds 8.3 33.3 50.0 15.4 

13-year-olds 11.1 22.2 50.0 3014 

17-year-olds, 16.7 14.3 40.0 44.4 

All ages 11.9 19.2 44.7 31.5 

Atlj usted ' 

9-year-olds 0.0 0.0 75.0 20.0 

13-year-olds 0.0 ' 0.0 57.1 . 30.8 

17-year-olds 16.7 15.4 45.0 47.4 

All ages 4.8 - 10.0 52.6 33.3 



30 



91 



Table 3-16. Conditional Percent of Linear Model Design Based (DB) Tests Versus Alternative Tests (AT) 
Reaching an Opposite Conclusion for Mean Scores 



Cross- Alternate Rejected by AT Accepted by DB Accepted by AT Rejected by 

Classification Test given accepted given rejected given rejected DB given 

by DB ■ by AT by DB accepted by AT 



Race^'^Sex'^PARED 



SRS 

9-yea r-olds 0 . 0 

13-year-"ol<js 0.0 

17-year-olds 0.0 

All ages 0.0 



I 

CD 
U) 

I 



Adjusted 

9-year-olds 0.0 

13-yea^r-olds 0.0 

17-year-olds 0.0 

All ages 0.0 



Sex»TOC-PARED 



SRS 

Q-yea r-olds 
13-year-olds 
17-year-olds 
All ages 



0.0 
20.0 
12.5 



92 



Adjus ted 

9-'year-olds 
13-year-olds 0.0 
17-year-olds. 0.0 
Ail ages 0.0 



0.0 
0.0 
0.0 
0.0 



60.0 
0.0 
33.3 
4&.0 



50.0 
0.0 
16.7 
22.2 



0.0 
0.0 
0.0 
0.0 



'60.0 
0.0 
33.3 
40.0 



50.0 
0.0 
16.7 
22.2 



0.0 
0.0 
33.3 
9.1 



50.0 
20.0 
33.3 
37.5 



100.0 
25.0 
20.0 
46.2 



0.0 
0.0 
0.0 
0.0 



87.^5 
60.0 
'33 . 3 
68.8 



100.0 
50.0 
16.7 
57.9 



conservatism 'observed for the SRS test is exaggerated fgr the adjusted 

•i . - 

test. This is a further reflection of the previous observation that the 
mea design effect is too large of an adjustment to divide the SRS test 
statistic by. The same observations can be made for the contrasts of NAEP 
mean scores (Table 3-14>. However, the results are less dramatic. In 
addition Tables 3-15 and 3-16 present the results for the r linear model 
based tests. Because of the deficiencies presented previously for this 
mode oi analysis, these two tables are presented for completeness^ only. 
3.3.2 Balanced Effects * * 

As noted in section 3.1, the balanced effects analysis proceeded by 
fitting two different linear models to the data and then assessing the'' 
significance of each term ;n the model 'after accounting from the remaining 
terms. These tests- are preseated in Appendix C. As was done for the Wald 
statistic/weighted least square data, contingency tables were formed of the 
number of tests which were either accepted or rejected at the five percent 
significance level by the sample design based test versus either the sampl- 
ing weighted standard regression test or the unweighted standard regression 
test. Again, the contingency tables were further summarized to yield Table 
3-17. This table presents the four conditional percents of reaching an 
opposite, conclusion for each contingency table. The first column of this 
table indicates that both of ,the non-sample design based testing procedures 
are far too liberal. These two procedures tend to reject about 20 percent 
too often. 




Table 3-17. Conditional Percent of Balanced Effects Design Based (DB) Test 
Versus Alternative Tests (AT) Reaching an Opposite Conclusion 



Rejected by AT Accepted by DB Accepted by AT Rejected by • 
Alternate given accepted given rejected given rejected DB given 

Test by DB by AT by DB accepted by AT 

Unweighted 

9-year-olds 17.6 21-^.4 15.4 12.5' 

13-year-olds 22.2 ,08.7 00.0 00.0 

17-year-olds 08.3 05.3 00.0 00.0 

All ages 15.8 10.7 03.8 05.9 

Weighted « 

9-year-olds 23.5 25.0 07.7 07.1 

13-year-olds 22.2 08.7 00.0 00.0 

17-year-olds 25.0 15.0 05.6 ^ 10.0 

All -ages - 23.7 15.3 03.8 . 06,5 
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4. COMMENTS ON NAEP DATA AND DOCUMENTATION 

Two main problems were observed with the data or documentation. First 
of all, the documentation contains an extensive description of the NAEP 
sample design and indicates that this design should be considered when 
analyzing the data. Unfortunately, the documentation does not indicate how 

this design is reflected in the data. The variable ISVARES is listed as 

•'3' 

the variance estimation code, but no indication is given as how to interpret 
this variable. Since this work was done at RTI, we were able to determine 
how this variable relates to the sample design, e.g. strata and primary 
sampling units. The second item that we would have found useful was a 
machine readable key for scoring the exercises. 
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Appendix A 
NAEP Exercises 



A-1 

99 



ERIC 



22. 







120-= 






i-iio 


ioo4 






1-90 


80-= 




60-= 


i-70 




r- 50 


40-= 






i"30 


20 -| 






|- 10 


0-| 






r— 10 


-20-^ 





What temperature is shown on this thermometer? 



■10° 
■ 5° 

5° 
10° 



ciD I don't know. 



Age Class 1 
Package 2 

Variable Name: N0222A 
NAEP No, : 5-D^1322 
Content Area: Measurement 
Unweighted Percent Correct: 



79,09 




CID 



STOP 



DO NOT CONTINUE 
UNTIL TOLD TO DO SO. 



loo 

A-2 



E 



I 



I: 




r 



EKLC 



-27. 



CID 



CID 
CZD 



CZD 



6000 



Which one of the following is CLOSEST to the number that goes in the box? 



aooo 

2000 
3000 
c=D 5000 



I don't know. 



Age Class 1 ' 
Package 2 

Variable Name N0227A 
NAEP No. : 5^B22745 

Content Area: Variables and relationships 
Unweighted Percent Correct: 23,14 



101 




eT/M»l ^^'^ CONTiNlir, 
Oldr I UNTIL TOLD lO DO SO 



:< \f^2nr, ill!) 1.'^ .i 



A-3 



5. A. Which is longer? 



c=D 2 feet 
( — ) 1 yard 

( — ) I don't know. 

B. Which is heavier? 

cz) 1-7 ounces 
CZD 1 pound 

ciD I don't know, 

C. Which holds more water? 

CZD 3 pints 
CUD 2 quarts 

CZD I don't know. 



Age Class 1 
Package 3 

Variable Name: N0305C 
NAEP No. : 5-E10003 
Content Area: Other Topics 
Unweighted Percent 
Correct: 79.98 



C2D 
CZD 



CUD 



CID 
CtD 

5-01081 1-429.1.2.3 

f;;Riiinn;M.2.3.4 




STOP 



DO NOT CONTINUE 
UNTIL TOLD TO DO 



A-4 



Sarah paid $1.20 for 6 bottles of cola including the bottle deposit. If the 
deposit on each bottle is 5 cents what is the cost of each bottle of cola? 



ANSWER 



Age^Class 1 
Package 3 

Variable Name N0317A 
NAEP No. : 5-A60942 

Content Area: Niimbers and Numeration 
Unweighted Percent Correct: 1.80 



CO 


COCO 


CD 




CO 


coco 


CID 




CZD - 


CO 


CO 


CO 


CO 


CO 


CO 


CO 


CO 


• CO 


CO 


CO 



S.AS0942-319-I.2,3 



A-5 




UNTIL TOLD TO DO SO. 



23. A. Which figure is OPEN? 




Age Class 1 
Package 3 

Variable Name: ;N0323A 

NAEP- No. : 5-C12411 

Content Area: « Shape, Size and 

Position 
Unweighted Percent 
Correct: 95.37 




I don't know. 



Which figure is CLOSED? 



c 



6.C1241 1.919.1 



D 

e 



I. don't know. 




STOP 



DO NOT CONTINUE 
UNTIL TOLD TO DO SO. 



CID 
CO 



0 



0. . ■ ' ;.;;»()0 + Q = 6000 

Which one of the followinL- is CLOSEST to the number that goes in the box? 



CZD 

CO 
CO 



CO 



1000 
CO 2000 
CO 3000 
CO 5000 



I don't know. 



Age Class 2 
Package 1 

Variable Name: T0105A 
NAEP No, : 5-B22745 

Content Area: Variables and Relatjonships 
Unweighted Percent Correct : 64,53 



I ^ .CO 

EWC '-^ ' A-7 




■ 1 n H ^rrso I not continue 

J-UO|%>IWr]^ UNTIL TOLD TO DO SO. 



What is the weight of this bag of candy? 
CD 225 g 

CD more than 225 g 
CZD less than 225 g 



I don't know. 



Age Class 2 
Package 1 

Variable Name: TOllOA 
NAEP No. : 5-D20922 
Content Area: Measurement 
Unweighted Percent Correct: 72.54 



CO 
dD 
dD 
CID 



ERIC 



A- 8 




STOP 



DO NOT CONTINUE 
UNTIL TOLD TO DO SO. 



3. What does j of 9 equal? 




Age Class 2 
Package 2 

Variable Name: T0203A 
NAPE No. : 5-C20006 

Content Area: Shape, Size and Position 
Unweighted Percent Correct: 47.72 



ERIC 



GD 
CZD 
CID 
CO 
C3D 

CZD 
GD 
CO 

-.CD 



COCID 



/ 



CO 
CO 
CO 



-^07 (stop 



1)0 NOT CONTlNUr. 

L NTIL TOLD TO 1)0 SO. 



1 , 
Ti' 



A- 9 



23. Kate averages 10 miles per hour on her bike. At this rate how far v^iU she 
travel in 5 hours? 



at- 



czD 2 miles 

5 miles . • , 
15 miles ' . 

50 miles 

More information is needed to solve this problem. 



I don't know. 



Age Class 2 
Package 2 

Variable Name: T0223A 
NAEP No. : ..5-E20941 
Content Area: Other Topics 
Unw€..lghted Percer.t Correct: > 86.27 



CO 



CZD 
' CD 
CO '* 



^08 

A-IO 




Crrstkl '^^ ^'^'^ CONTINUE 
5TOr B UNTIL TOLD TO DO SO. 



CO 



Sarah paid $1.20 for 6 bottles of cola including the bottle deposit. If the 
deposit on each bottle is 5 cents what is the cost of each bottle of cola? 



ANSWER 



Age Class^2 
Package 2 

Variable Name: T0224A 
NAEP No. : 5-A60942 

Content Area: Numbers and Numeration 
Unweighted Percent Correct: 22.24 



5- A 609 1 :i 9IIM 




-eT^Bl '^0 NOT CONTINUE 
OTUr i UNTie'TOLD TO DO SO. 



o 



6. A car traveled eight kilometers in five minutes. At this speed, how many 
KILOMETERS could it travel in one hour? 



ANSWER 



Age Class 3 
Package 2 

Variable Name: SO206A 
NAPE No: 5-C50014 

Content Area: Shape, Size and Position 
Unweighted Percent Correct: 54.40 




CO 
CO 

CO 
CXD 
CO 

CO 
CO 

5-092512- 




CO 
OD 
CO 
CO 
CD 



A-12 . 




STOP 



DO NOT CONTINUE 
UNTIL TOLD TO DO SO 



E 



E 

E 
t 

E 
£ 

E 
E 



ERIC 



25. 



CO 



CO 




Sarah paid $1.20 for 6 bottles of cola including the bottle deposit. If the 
"^epostrorTeacH^blotTle^ 



ANSWER 



Age Class 3 
Package 2 

Variable Name: - S0225A 
NAEP No. : 5-A60942 

Content Area: Numbers and Numeration 
Unweighted Percent Correct: 44.06 



(HDCID 



CO 
CO 



CO 
CO 



.CO 
CO 
CO 



111 




STOP 



DO NOT CONTINUE 
' U^!'JIL TOLD TO DO SO. 



A-13 



Betty set the timer to run for 40 minutes. It had run for 15 minutes when she 



discovered it should have been set to run for 50 minutes. Where should she 
reset-the timer now to correct the mistake? ^« 



ANSWER 



Age Class 3 
Package 1 

Variable Namer S0108A 
NAEP No: 5-D94043 
Content Area: Measurement 
Unweighted Percent Correct: 61.29 



C2D 
CID 




CD 


CID CID 
CZDCID 


CO 


QDGD 


CZD 


CID 


CID . 


CO 


dD 


CO 


dD ' 


CO 


ao 


CO 


CD 


CO 



il2 



A-14 




STOP 



DO NOT CONTINUE 
UNTIL TOLD TO DO SO. 



V 

I 
E 



t 

E 
6 

E 



ERIC 



17. 



3900 + = 6000 ■ ' 

Which one of the following is CLOSEST to the number.that goes in the box? 
c=3 1000 

c=3 2000 ^ 

cz) 3000 
czz) 5000 

CZ) I don't know. 



Age Class 3 
Package 1 

Variable Name: S0117A 
NAEP No. : 5-B227^5 

Content Area: Variables and Relationships 
Unweighted Percent Correct: 85.66 



GD 
CD 
CZD 
CZD 
CD 
CD 

CZD 
CD 
CD 

.5'B'm45'9l»'l/J.:l 



113 




STOP 



DO NOT CONTINUE 
UNTIL TOLD TO DO SO. 



A-15 



21. The lost dog is small and black, 

4 

.A. It I see a small brown dog, then 



it might be the lost dog. 

it must be the lost dog. 

it could not be the lost dog. 



I don't know. 



B. If I see a small, black'dog, then 



it might be the lost dog. 

it must be the lost dog. 

it could not be the lost dog. 



I don't know. 



Agie Class 3 
Package 1 

Variable Name: S0121A 
^ ^NAEP No. : 5-E50248 

.Content Area: Other Topics 
' Unweighted Percent Correct: 95.82 



ERLC 



CO 



GD 
CO 

CO ' , 

.7'lr:50248'9l»-l/-'.:i 




STOP 



DO NOT CONTINUE 
UNTIL TOLD TO DO SO. 
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Appendix B 
Wald Statistic Chi Squareds 



1.15 



ERIC 



I ■ 



Table B-1- NAEP Item Wald Statistic Chi Squareds for the Race*Sex*PARED 
Cross-Classification 



CHI SQUAREDS 



SIGNIFICANCE LEVELS 



MA CP 




IT r M 


f C o 1 


MUM H r R 


NUM RCR 


N ' ? 2 2 A 


t 

i 


N " 2 2 2 A 


2 


M ' O 0 O A 
N . £ ^ c A 


■I 


N '*2 2 2 A 




N »2 2 2 A 


5 


N '*2 2 2 A 




fj ^ 0 0 0 A 
H C t C M 


7 


f .' 2 2 ? A 


a 


N ' 2 2 2 A 


9 


N 12 22 A 


1 0 


N 2 2 2 A 


\ \ 


W' 2 2 2 A 


i ? 


N ' 2 2 2 A 


1 3 


N ' 2 2 2 A 


1 4 


fi * 2 2 2 A 


1 5 


N 2 2 2 A 


I 6 


N * 2 2 2 A 


^ 7 


N '2 2 7 A 


1 


N ' 2 2 7 A 


2 


*t ' O 0 1 k 


3 


W ' O 0 T h 




M 5 5 7 A 


J 


N " 2 2 7 A 


D 


M , 2 2 7 A 


7 




a 

n 


N 2 2 7 A 


Q 


1^ ^2 2 7 A 


1 I- 




1 1 


N r 2 2 7 A 


1 o 
I c 


N 2 2 7 A 




i\ 'cell* 


1 A 
i 


hi O O T A 


1 s 
i 3 


M "1 0 7 A 


4 o 


M r O O T A 


1 r 


N C 3 Q 5 C 


1 


N r 3 0 5 C 


o 


IN JO li J L 




fcl ft /, c /• 




N 3 0 5 C 


5 


N .3:5C 


6 


n:305C 


7 


N:»3C5C 


8 


N '3C5C 


9 


N:3C'?iC 


1 0 


N:305C 


1 1 


rr305C 


. 12 


n:305C 


i 3 


N'''305C 


1 4 


N^'srsc 


15 ■ 


N;305C 


' 16 


N:305C 


17 



D.F. 



OE S I GN 






DESIGN 






DASED 


SRS 


ADJUSTED 


BASED 


SRS 


ADJUSTED 


8*49 


6.76 


4.55 


.0752 


.1491 


• 3363 


0 • 12 


0.10 


C. '^7 


. 7298 


.7512 


.7947 


I • C4 


0.21- 


0.14 


. 3C85 


.S477 


,7077 


2,20 


1 . 89 


1.2 8 


.1377 


.1687 


.2587 


0.45 


1 . 32 


0.89 


.5C2 7 


.25C3 


.3454 


2 • 06 


0. 09 


C.C6 


.1514 


.7653 


.8065 


4,72 


0 . 82 


0.55 


.1)299 


.365r) 


.4573 


•J .32 


1 . 76 


1.18 


.5691 


.1649 


.2767 


94 • 83 


17.58 


11.84 


.OQUO 


.0919 


.3760 


0.57 


0.99 


0.67 


.4494 


.3201 


.4145 


r,« :)9 


C. 16 


0.1 1 


.7632 


.6849 


.7391 


7.12 


5.28 


3.55 


.'J285 


.0714 


.1691 


1 .64 


1.43 


C.94 


.2QC6 


.2373 


.3322 


9.31 


6.92 


- 4.66 


.1570 


.3282 


.588(; 


7 • 36 


4.16 


2.8C 


.2885 


.6549 


.8333 


21.77 


11.41 


7.68 


.0054 


.1 797 


.4651 


8« 49 


6. 76 


4.55 


. 0752 


.1491 


.3363 


4 • 05 


,3.02 


2.16 


.3987 


.5539 


.7059 


3*68 


3.93 


2.81 


.r»5 5 0 


.3474 


.0936 


0 • t( 


0.73 


C .52 


.97 73 


.3933 


. 47 0^ 


6 8*34 


5 U . 77 


36.32 


. O^JCO 


. C 000 


. 000 j 


1,24 


C . 5'J 


0.36 


.26 6 3 


.4808 


.5511 


0 • 21 


0.03 


CO 3 


.6439 


.9944 


. 9953 ' 


Oi* 35 


C! . 05 


0.0 4 


.5535 


.8158 


. 84 3B 


1 • 80 


0.54 


U . 3 9 


.17 99 


.4618 


.5337 


598 4 2 2 


13 8.12 


98.7 9 


. UO G 0 


. OUOO 


. OObU 


15 • 29 


2 4.44 


17.48 


.0001. 


.3000 


. 00 oo 


2*11 


2.45 


1.75 


.1466 


.1178 


.1859 


AS- SA 


4 A . X A 


33.17 


.0000 


, 0000 


. 00 00 


u • C> J 


U . O H 


0.46 


.439 0 


.4 2 32 


.4982 


5 9-7 ii 
£ r • f V 




2C. 1 2 


" ,0309 


.3001 


.00 26 


1 U • 15 


5 . 05 


3.61 


.11-85 


.5379 


. 7294 


1 4 8 • 85 


5 4.76 


39.16 


.000 0 


. OOOU 


. 0000 


4 • 'j5 


3.32 


2.16 


.398 7 


.5539 


.7 059 


1 <J • O A 


16.47 


12.21 


. 0 0 7 9 


.002 4 ■ 


. 01 58 


2 • 13 


0.66 


0.4 9 ' 


.1445 


.4149 


.4826 


1 •33 


0.23 


0.17 


.2488 


.6307 


.6789 


t) • C6 


1.20 


C .89 


.7996 


.2724 


. 34 46 


C.48 


D.30 


C.22 


.4878 . 


.5857 


.6388 


n,3i 


2.91 


2.16 


.5791 


.C880 


.1413 


1.11 


0.10 


0.07 


.2914 


.7549 


.7881 


0 • 73- 


0.37 


0.28 


.3935 ^ 


.54 08 


.598^ 


14 0. 11' 


106.54 


79.01 


. 0009 


.0000 


. 0000 


V 13.49 


26.62 


19.74 


.0002 


. 0000 


. 0000 


C.44 


0.58 


0.43 


.5088 


.4457 


.5113 


2.99 


7.18 


5.33 


.2237 


.0276 


.0698 


l.BO 


3.05 


2.26 


.1794 


.0306 


.1325 


87».93 


7C.44 


52.24 


.0000 


.0000 


.0000 


3.97 


2.44 


1 . 8 1 


.6812 


.3749 


.9362 


17.00 


2 7.92 


20,71 


.0301 


.0005 . 


.0380 


13.81 


16.47 • 


12.'2I 


. 0079 


. 0024 


.0158 















Table B-1. (continued) 



CHI SQUARCDS 



SIGNIFICANCE LCVCLS 



NAEP 
ITEM 

NUMDER 



TflST 
NUMBER 



D.F 



ERLC 



N'323A 


) 


N . 3 23 A 


2 ° 


H .3 2 3 A 


3 


N 3 23 A 




N ' 3 2 3 A 


5 


N .323 A 


6 


N 323 A 


7 


N *■ 3 2 3 A 


B 


N .323A 


9 


N - 3 2 3 A 


1 0 


N 3 23 A 


1 1 


N '323 A 


1 2 


n •'3 23 A 


1 3 


^^•^3 23 A 


1 4 


N 3 2 3 A- ' 


1 5 


W ' 3 2 3 A 


1 6 


N ' 323 A 


1 7 


T (.1 05 A 


f 


T ' 1 ' 5 A 


2 


T .'1 ?5 A 


3 


T 1 C 5 A 


i» 


T ; 1 * 5 A 


5 


T ' 1 U5 A 




T 1 05 A 


7 


It 1 05 A 


g 


T r 1 0 5 A 


9 


T ')1 ')5 A 


1 (' 


T > 1 C. 5 A 


1 1 


T '1 Gf> A 


1 2 


T '.IPSA 


1 3 


T ''I 0 5 A 


1 4 


T '-^ 1 C 5 A 


1 5 


T C 1 '.5 A 


1 6 


T n 05 A 


1 7 


^ l1 1 0 A 


1 


y^'llOA 




T'mA 


3 


T 0 1 1 0 A 




T .llOA 


5 


T 111 A 
1 s' 1 1 «« A 


o 


T : 1 1 0 A 


7 


If 11 OA 


8 


T'^llCA 


9 


T'U 1 "JA 


1 u 


T i i : A 


1 1 


TMl.OA 


12 


T r 11 OA 


13 


iri IC A 


14 


t:»ii:a 


15 


T:U10A 


16 


T-'llOA 


1 7 









DES I GN 






DESIGN 






f • 


CASED 


SRS 


ADJUSTED 


BASED 


SrS 


AOJUSTE 




1*76 


1 *90 


1*66 


*7805 


• 7549 


'.7983 




10*67 


11*24 


9*85 


*( Oil 


• 0008 


.U017 




0*79 


0*12 


0*1 C 


*i751 


• 7294 


. 7462 




6*14 


6*17 


5*40 


*C132 


• C 1 3 0 


. 02 01 




G* 29 


C* 08 


0* 07 


*5919 


• 7824 


. 7961 




10*11 


6*80 


5*95 


*0015 


• UU91 


. 0147 




0*22 


0* 00 


0*00 


*6364 


• 989Q 


.9897 




0 0*27 


0*08 


0*07 


*6032 


.7775 


.7914 




22*46 


26*48 


23«19 


• 0211 


• 0055 


. 016S 




< ■•6*51 


13*10 


11*47 


* 01 07 


• 00C3 


. DU07 




n* 47 


ri*49 


C*43 


*4925 


• 4849 


.5134 


2 


6*2"^ 


7*03 


6*16 


.n436 


• 0298 


. 0461 


1 


c* ro 


0*00 


0*00 


.9639 


• 9743 


.9759 


6 


1 1 *64 


1 6*64 


1'4*57 


. 07 04. 


• 0107 


. 0239 


6 


5 *52 


^5 * 37 


4*70 


• 4789 


• 4977 


.583'J 


8 


11 *97 ' 


9*31 


8*15 


.1527 


• 3166 


.4186 




1* 76 


1«90 


1*66 


«,78C5 


• 7549 


.798j 


<j 


5*60 


6*73 


4*14 


• 2143 


• 1509 


.3873 




10*40 


8*51 


5*2 3 


• 0013 


. 0035 


. 0222 




C * 15 


0 * C6 


U*03 


• 7(;U9 


• 81,39 


.8533 




4 8 * 51 


56*09 


34.48 


• 0 0 0,0 


• 0 U 0 1 


• 0030 




C* 03 


0*01 


0.01 


• 8639 


• 9224 


.9391 




0 * ni 


Call 


0*07 


• 9254 


• 7448 


.7985 




C *23 


C * 12 


0*38 


• 62 66 


• 7253 


• 7829 




G * 151 


0*11 


C * 0 7 


• 93 19 


• 7347 


. 7905 




314*21 


179* 88 


110*57 


• 0000 


• UQ03 


• OOOC 




24*47 


45*21 


27*79 


• c-auo 


• OGOO 


. CjlOO 


1 


1 * 24 


2*44 


- 1*50 


• 2663 


• 1179 


.2202 


2 


3 9*22 


5 8*07 


35*70 


• 0 a p 0 


• 3000 


. 00C3 




1*05 


2 * 04 


. 1*25 


• 3059 


• 1535 


. 2631 


6 


114 * 55 


4 9*98 


30*72 


• GOOO 


• 00C3 


. OOOU 


6 


6 .83 


7.41 


4*56 


• 3373 


• 2844 


.6018 


8 


91*33 


82. 79 


50*89 


• 0000 


• 0000 


. 0000 




5*80 


6 * 73 


4*14 0 


• 2143 


• 15C9 


. 3873 


^ 


4*52 


6*83 


4*79 


• 34 04 


• 1451 


.3097 




15*42 


2 6 *54 


18*60 


• 0001 


• 0003 


.0000 




20*95 " 


3.32 


2*33 


• 0000 


• 0682 


• 1269 




29* 37 


25*14 


17*62 


• 0000 


• 0000 


• 0000 




14*10 


4.20 


2*95 


• C 0 0 2 


• 0404 


• 0861 


1 


8*67 


1 0*69 


7*49 * 


• 0C32 


• 0011 


• 0062 




3 0* 87 


8* C6 


5*65 


• 00 0 0 


• 0 045 


• 01 75 




14* C7 


5*64 


3*95 


• 5302 


• 0176 


• 0469 


LI . e 


97*^8 


96*54 


67*68 


• 0000 


• ouoo 


• 0000 




2 0*57 


44 * 85 


31*44 


• GOOO 


• OOCO 


• OOOC 




2*77 


2*64 


1*85 


• 0959 


• 1042 


• 1737 




14*21 


21*31 


14*94 


• C008 


.0000 


• 0U06 




0*21 


0*25 


•0.17 


•6466 4 


.6179 


.6762 




45*16 


54*61 


3R*2R 


• OOCO 


.0000 


.GOOO 


6 


4 4 * 39 


24*78 


17*37 


• 0000 


. 0034 


. 0080 


a 


70*17 


43*90 


30*77 


• OOOO 


. 0000 


.30G2 


4 


4*52 


6*83 


4*79 


• 34 04 


• 1451 


.3097 








11? 









Table B-1. 



(continued) 



NAEP 



ITEM 


VC ST 




NUMBER 


NUMBER" 


D • 


T ""Z 33 A 


1 




T , 2,03 A 
T ( 2^^ 3 A 


2 




3 




T r 2 0 3 A 






T ' 2 05 A 


5 


1 


T 2 Z3 A 


6 




T ^2 j3 A 


7 




T ' 2 0 3 A' 


B 




1 2 0 3 A 


9 


1 1 


T 2 0 3 A 


1 ' 




T ^ 2 C5 A 


I 1 




T '20 3 A 


1 2 


2 


T '2 i3 A 


\ 3 


1 


T 2 0 3 A 


1 4 


6 


T . 2 0 3 A 


1 5 


b 


T C 2 3 3 A 


1 & 




T 2 03 A 


1 7 




T ^223 A 


1 




T :223A 


2 




T ^^223 A 


3 




T » 2 2 3 A 


<» 




T '2 2 3 A^ 


5 




T ''223a'' 


6 




T * 2 2 3 A 


7 




T'"223A 


8 




T '223 A 


9 


11 


^ T r 2 2 3 A 


I 0 




T ''223A 


1 1 




T 223A 


1 2 


2 


T * 2 2 3 A 


\ 3 


^ 


T 223A 




6 


T 0223A 


1 5 


6 


T :-2 23 A 


1 6 


6 


'T *1J 2 3 A 


I 7 




T ' 224 A 


1 




T :224 A 


2 




T '2 24 A 


3 




T"224 A 






T- 224A 


5 




TC22<»A 


6 




T ' 224 A 


7 




T t2 2 4 A 


8 




To224A 


9 




Tn224A 


10 




TC224 A 


1 1 




Tr224A 


12 


2 


TC^a-^A 


13 


1 


T0224A 




6 


T&224A 


1 5 


6 


T(224A 


16 


8 


T j2 24A 


V 17 


4 



CHI SOUAREDS 



DESIGN 



BASED 


S R S 


An 1 1 f c T r n 
A U J U o 1 L U 


u • 97 


1 ikC 




1 • b4 


1 T t 

1 • Ob 


ft Q 9 


5*72 


4.80 


9 on 


i J • J c 


T n UQ 


1 A 6 A 
1 O . f u 


£.7 

J • D^r 


9 £.0 
c • oO 


1 £.9 
1 . b £ 




9 £.1*1 




2*23 




1.09 


n • 46 




U . 0 3 


394 • 74 


9 n Q 9 4 


1 9 £ 9 7 
1 £ D • £ r 


0 Lt • □ r 


4 9.39 


9 Q Q 


3 • 59 


7 Aft 


A A 7 








U • £ H 


if . 0 b 


A 9 1 


4 J • 01 


(1 A 9 7 


■ 5 0.85 


4 3 • 4 2 


4 8.40 


9 Q 9 1 
£ 7 . £ 1 


5 3*02 


R A R 7 


T 9 n T 


C • 97 


1 A £. 

1.4b 


U . o 0 


11* 92 


8.63 


£. 7 9 

b . r £ 


2*13 


1.05 


0.82 


r • 02 


4.89 


3.81 


4*03 


5.46 


4.26 


0 • 59 


5.20 


4.05 


1 • 4 9 


U . 9 1 


rt 7 1 


C • 1 0 


D . b J 


4.36 


0 • 57 


5.32 


4.14 


61*54 


6 9.13 


5 3.86 


' 14*63 


2 3.85 


18.58 


0 ■ 20 


A 9 7 


ti 9 1 
U . £ 1 


8*18 


12.11 


Q AT 


3 • 98 


0 . b 1 


2.81 


3 7*66 


4 1.91 


3 2.65 


9 • 1 7 


C 7 7 

b . r r 




2 4*34 


2 9.27 


2 2.80 


11*92 


8.63 


6.72 


10*67 


8.51 


7 7 


0*09 


0.19 


It 


C • C3 


, ,C • 4 0 


0.27 


1 8 . 84 


2 2.91 


15.55 


' ^ ft 
L> • i U 


1 . bo 


1 19 
1 . 1 £ 


9.05 


9.23 


6.24 


0 . U5 


' 1.70 


1.15 


1 .93 


2.04 


1.38 


245.65 


201.04 


136.43 


29.35 


70.86 


48.09 


0.13 


0.27 


0.18 


21.37 


23.54 


15.97 


0.60 


U .68 


0.4 6 


75.95 


115.41 


78.32 


15.22 


15.44 


10.48 


76.96 


50.40 


' 34.20 


10.67 


8»5X 


' 5.77 



SIGNIFICANCE LEVELS 
DESIGN 

BASED SRS ADJUSTED 



.9149 


.8332 


• 92 7? 


.2150 


.2432 


.3646 


. (1 1 6 8 


. 0265 


• 0889 


. 0303 


.0030 


. do 00 


.055 4 


.1015 


. 20 33 


.1199 


.1069 


.2104 


.13 51 


. 1 799 


. 29 75 


. 4962 


.8261 


.8645 


. (' U 0 3 


. (J U 0 3 


,00 00 


• 000 0 


. C 0 0 0 


. ooon 


.0582 


.0065 


. 0 3 4 G 


c n n Q 
. u U U 7 


n fj 11 fi 
. u u u u 


. 0001 


£.999 
. D £ £ £ 


. 3 3 J 7 


.6431 


.0000 


. 3 U 0 0 


. 0 0 0 11 


n ft fi ft 

m V yj M V 


n ft 1 *j 

. u U U w 


. 0001 


r n n '1 

m Vv St J 


. 0300 


. C 0 01 


Q 1 A Q 


. 0 0 0 £ 


.92 70 


. U 1 u '1 


ft 7 1 1 


.13 10 


1 A A ft 
. 1 H 4 U 


T ft R A 
.,0 'J 3 4 


T£, 
. Ob 39 


.8958 


19 7 1 
. J £ r 1 


Ac; 1 ft 
. (13 1 V 


n A A £. 
. U 4 4 o 


ft 1 Q A 
. U 1 7 4 


ft T 0 1 
. U 0 ' 1 


. H H J 3 


9 9 £. 
. U £ £ b 


n4 4 1 

. W ^ 1 1 


9 9 9 ^» 
. £ £ £ D 


T T Q Q 


• 3 9 95 


7 R ft 7 
. r 3 U f 


ft 1 7Q 
. U 1 r 7 


n T £. 7 
. UO D f 


A A Q R ■ .>^^ 
.4473 


A 9 1 1 
. U £ 1 1 


ftA 1 a 

• U4 I O 


M il ft •^ 


ft rt m 


.0009 


n n 1 

. U i> U 1 


n ft ft ft 


. 0 0 0 c 


. 3 7 C3 r 


.6007 


.6441 


. 'J 1 D r 


.0023 


. 00 89 


<1 A £. 1 

. U4 b 1 


ft S7 4 
. u 3 r H 


AQ T 

. U 7 o 9 


rt n A ft 
. u u u u 


n ft 0 p 

. U U U u 


Aft nr 

. u U U V 


1 £, A T 
. 1 b 4 0 


T A 9 Q 
. 0 4 £ 7 


. 5393 


ft rt 9 ft 
. U U £ U 


fi n ft T 

. U U U 0 


ft ft T £. 

. w U 0 b 


^ 1 fl ft 


ft 7 1 1 


1 R 1 T 


. u J u b 


ft 7 4 ^ 


.2167 


7^ 4 A 
. r b ^ 0 


.6616 


.7184 


.8694 


.5272 


.6024 


.0000 


.couo 


.0001 


.7576 


.1983 


.2893 


.0017 , 


.0324 


.0125 


.6234 


.1926 


• 2831 


.1649 


.1532 


• 2393 


. 0003 


. 0000 


• OOCO 


.OCJO 


.0000 


• 300d 


.7142 


.6050 


• 6701 


.0000 


.0000 


.0003 


.4395 


.4104' 


.4977 


. 0000 


.0000 


.0000 


.0186 


.0171 


. 1058 


.0000 


oOOOO 


.OOOC 


0 .0306 


• 0746 


• 2167 



Table (continued) 



NAEP 



ITCM 


TEST 




DESIGN 


in V/ n L> w r\ 




O.F. 


BASED 


S 1 G B A 


1 




7.65 


S I'l C8 A 


2 




3n.47 


S 


3 




1 . 14 


S ;1*}8A 






27.46 


S X C 8 A 


5 


J 


0.C3 


S ' 1 08 A 






2.25 


S ' 1 Utt A 


7 




n.?o 


S ' 1 B A 


8. . 




:.45 


s :. 1 0 8 A 




11 - 


154.91 


Sf>l 08A 


1 0 




51.82 


5 • 1 C 8 A 


1 1 


I 


5.15 


S ■ 1 J 8 A 


1 2 


2 ^ 


27.38 


S .* 1 0 8 A 


1 3 


J • 


0.22 


1 C8 A 


1 4 




73.31 


S ' 1 08 A 


15 




9.33 


s • 1 r 8 A 


1 6 




46.t>9 


S ' 1 08 A 


1 7 




7.65 


S ij 1 1 7 A 


1 




9.55 


S " 1 1 7 A 


2 




5. 03 


S *« 1 1 7 A 


3 




2.74 


S ' 1 1 7 A 






. 9.13 


S ^ 1 1 7 A 






3. 34 


S C 1 1 7 A 


6 




r. 01 


S ' 1 1 7 A 


7 




2 .56 


S * 1 1 7 A 


a 




2.81 


S *1 1 7 A 


9 


1 1 


136. 21 


s n 1 7 A 


' 1 f» 




2 6. 75 


S '1 1 7 A 


1 1 




C' . 03 


S f - 1 1 7 A 


1 2 


2 


1C.42 


s r 1 1 7 A 


1 3 


1 


0 . ai 


S 1 1 7 A 


1 4 




65. C7 


S ( 1 1 7 A 


\ 5 


6 


7 . r4 


S " 1 1 7 A 


1 6 


8 


4-5 .33 


S 0 1 1 7 A 


^ 17 




9.55 


S 112 1 A 


1 




1.01 


S ' 1 2 1 A 


2 




' r.ci 


S ^1 2 1 A 


3 




1.21 


s r 1 2 1 A 






1.47 


S " 1 2 1 A 


5 




0. 84 


Sn21 A 




1 


n. 16 


S "121 A 


7 




0.08 


Sri21 A 


B 




3.74 


S ^ 1 2 1 A 


9 


31 


21.48 


S'^ia^ A 


1 Q 




1.92 


S 1 4il 1 A 


1 1 




2.72 


S f. 1 2 1 A 


1 2 




1.70 


S.:i21 A 


1 3 




C.59 


S r. 1 2 1 A 


14 




6.45 


SC121A 


15 




9.45 


S J121A 


16 


8 


6.59 


0 !1 A 


17 


H 


l.Ul 



SQUAREDS SIGNIFICANCE LEVELS 

DESIGN 



SRS 


ADJUSTED 


BASED 


SRS 


ADJUSTE 


5.67 


5.12 


.1051 


.2255 


.2753 


14.63 


13.21 


.OOOU 


. 0001 


.0003 


0.16 


0.15 


.2852 


.6884 


.7031 


20.53 


18.54 


.OCtOO 


• 'J C U 0 


.COCO 


0.55 


0.05 


. 8688 


. 8227 


.831(» 


& . a6 


0.06 


.1337 


.7991 


.8089 


0.19 


0.17 


.9494 


^ ';663n 


.6787 


0.30 


0.27 


.5036 


.5829 


.6017 


iec.7i 


163.23 


.COCO 


.0000 


^COOO 


77.47 


69.98 


.OUCO 


.0000 


. ooop 


4.23 


3.8 2 


. 0233 


.0397 


. 05 OS 


23.28 


2 1 .U3 


.0000 


.OOOU 


. 3 0 C fJ 


0.12 


U.ll 


.6363 


.7255 ' 


. 73 86 


87.82 


. 79.32 


.DCOO 


. 0000 


.UOOti 


7.87 


7.11 


.1557 


.2480 


.3111 


34.49 


3^1 . 1 5 


.(TO DO 


.0000 


.OOCl 


5.67 


5.12 


. 1L51 


.2255 


• 2753 


7. 08 


5.76 


.0466 


.1315 


.2180 


5.97 


4.85 


.0249 


.C146 


^0277 


1.41 


1.14 


..6977 


.2353 


• 2847 


6. 05 


4.92 


. 0025 


.3139 


• 0266 


Q.64 


0.52 


. 0676 


.4234 


.4705 


0. 05 


0.O4 


.9083 


.8264 


.8433 


1 .34 


I ; C» 9 


.1096 


.2471 


.2968 


0 . 85 


0.69 


.0935= 


.3577 


.4071 


9 1 .62 


74.45 


.OOOU 


.COOU 


.tiOUO 


4 2.60 


34.62 


.0000 


.0003 


.ocoo 


0.04 


0.03 


. 8709 


.8402 


.8558 


9.32 


7.57 


. U055 


.0095 


. 02 27 


0.02 


0.02 


.9134 


.8856 


.8963 


4 8.65 


39.53 


.0000 


«C000 


. 0.0 00 


3.75 


3.03 


.3169 


.7136 


.8052 


2 3-55 


19.13 


• odOo 


. ro27 


.0142 


7. 08 


5.76 


. ^0486 


.1315 


.2180 


2 .25 


1.65 


.9081 


.6895 


. 8002 


0 .33 


0.24 


.9103 


.5643 


.6219 


0.80 


0.59 


.2722 


.3700 


.4432 


2.50 


1.83 


' .2247 


. .1136 


.1759 


0.00 


0.00 


.5598 


.9485 


.9560 


0.02 


0.02 


.6d>7 


.8838 


.9005 


C.02 


0.02 


. 7837 


.8840 


.9007 


0.15 


0.11 


.U53U 


.6979 


.7399 


12.98 


9.50 


.0287 


.2945 


.5760 


2.28 


1.67 


. 1655 


.1310 


. 1965 


4.54 


3.32 . 


.0994 


. 0330 


. 0682 


3.05 


2.23 


• ^.275 


• 2176 


♦ 3276 


0.51 


0.38 


.44 06 


.4736 


.5399 


3.88 


2.84 


.3749 


.6934 


.8292 


7.44 


5.44 


.1496 


.2821 


.4884 


6.24 


. 4.56 


.5818 


.6205 


.8030 


2.25 


1.65 


.9081 


.6895 


.80 02 
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MAEP 
ITEM 
NUMBER 



TCST 
NUMBER 



D.F, 



CHI SQUAREDS 



DESIGN 
BASED 



SRS 



ADJUSTED 



SIGNIFICANCE LEVELS 



DESIGN 
BASED , 



SRS 



ADJUSTED 



S'2(>6A 

S:2C6A 
S '•2C6A 

s • 2 : 6 A 

5 ■ 2 x" 6 A 
S '2u6A 
S '206A 
S j2 06 a 
S')206 A 
S :2bb-A 
S-'?06A 
S' 216A 

s;2i:6 A 

S 'i2'06A 
S .2C6A 
S '206A 
S"206A 
S"225A 
S^225A 
S5I 225A 
Sr 225A 
S''225A 
SC225A 
s: 225A 
Si 225A 
S'225A 
S0225A 
SC.225A 
SC225A 
Sr225A 
Sf?225A 
SC225A 
Sr(22bA 
St 225A 



1 

.2 
3 

5 
6 
7 
B 
9. 
1 0 

: 1 
1 2 
i 3 
1 4 

15 
16 
I 7 

1 

2 

3 

5 
6 
7 
8 
9 
1 0 
1 1 
1 2 
1 3 
1 4 
15 
1 6 
1 7 



0.89 


1 . 07 


C .77 


.9257 


Q 0 Q T 

. 0 7 0 r 


• 94 3 1 


1 A « 4 3 


1 1 .52 


3.22 


. 0 C 0 1 


.0007 


• Q C 4 1 


5 « 66 


11.61 


8.28 


ft 1 "7 T 


.. U U u r 


• 004!) 


4 0 • A4 


,48.10 
0.45 


3 4.33 


. 00 0,3 


> .0003 


• 000? 


C • 25 


0.32 


£. 0 ^ 
. b £ V J 


.5031 


• 5 f ija 


1 • 65 


C . 98 


0.7? 


.1992 


.32 34 


• 4 0 4 1 


1*67 


3.82 


^^2 . 7 3 


.19 64 


. 0505 


• 09 85 


0 • 12 


0.33 


0.23 


.7269 


.5681 


• 62 97 


68 3 « 82 


4 ? 6 . 6 2 


2 90 • 21 


.0000 


fi r .1 1\ 


• 03 03 


7 J m (M 


116.97 


6 3.48 


.coco 


. 0 0 0 C 


• 0 0 C 'j 


1 3 « 




1 rt "TO 
lUm f C 


.0002 


ft ft fi 1 
. u U u 1 


• 0«) 1 1 


'■''•32 


4 7.25 


3 3 » 7 2 


.0000 


ft ft A 1 
• U U U J 


• (1 u w J 


K m JO 


0.16 


0.11 


.7825 ' 


£.01*1 
. b 9 1 1 


• r J b c 


1 3 1 « 82 


12 7.38 


9 0.91 


. 0 C 0 


ft ft ft 


• 0.0 0 D 


3 6 « 50 


2 3.76 


16.96 


' fl A ft ft 


n ft A £. 

. u U U b 


• W U 7 "t 


Ic c « s 1 


8 4.94 


6 0.-6 2 


P *i ft ■* 

. u □ I' u 


• 0 3 0 0 


• 0 0 0 0 


C • 89 


1.07 


0.77 


• 92 5 7 


. 8987 


• 94 3 1 


12*29 


1 0 « C6 


10.16 


.0153 


. 0395 


• 03 78 


1 « 70 


3.96 


U 1 


.1919 


. 04 b5 


• 04 5 4 


0 m 75 


0.52 


U . D J 


1 U (^ U 

. J 0 D 7 


.4698 


• 4 6 75 


2C ,49 


26.47 


26c74 


• 000.0 


.OOOC 


• oooa 


3.80 


2'. 05 


2.07 


.C513 


• 1522 


• 15C1 


,^.33 


2.60 


2.83 


.0039 


• 0943 


• 092S 


1 .73 


C°.62 


0.63 


.1883 


• 4300 


• 4277 


1 .21 


0.73 


0.74 


.2714 


• 3921 


• 389S 


516.97 


311.19 


314.43 


.caoc 


• 0003 


• 0003 


85.29 


91.22 


92.17 ' 


. 0000 


• 0000 


• 0000 


18« 13 


1 6.67 


16.84 


• f'QCO 


• 0000 


• 0000 


25.68 


26.72 


27.00 


.0000 


• coco 


• 0000 


6.'i5 


7.C8 


7.15 


.0 0,99 


• 0078 


• 0075 


10^.31 


113 .36 


114.55 


.00 00 


• 0003 


• 0003 


62.55 


2 6.63 


26,.9l 


.ocoo 


• 0002 


• 0002 


111.85 


57.14 


57.73 


. 0000 


• 0000 


• 00 30 


12.29 


1 0.06 


10.16 


.0153"* 


• 0395 


• 0373 



ERIC 



120 



Table B-2. NAEP Item Wald Statistic Chi Squareds or the Sex*TOC*PARED 
Cross-Classification 



NJ A r'c> 






f T r u 
1 ! L n 


TEST 




IJMBE R 


All IM R r 0 




M •• O O O A 

r J 2 c ^ * 


1 
i 


D 


M • O O O A 

N 2 2 2 A 


d 


1 
i 


N 2 2 2 A 




0 


N « 2 2 2 A 


A 


1 
1 


N 2 2 2 A 


c 
3 


£ 


N £ C. C m. 


D 


1 
i 


N ' 2 2 2 A 


7 


O 


M f O 1 O A 


a 
o 


2 


M *• 0 d 0 A 

n c. c. c. H 


u 




M S» 9 A 
rJ 1 c c c A 


i -J 




ft' O 0 0 h 
r> £ £ £ M 


1 1 


2 


AJ ' O 0 A 
iM c. C C.I*' 


1 9 


2 


fi ' O O 0 A 


i O 




nt C. £ Ht 


1 


9 


iK\ 'J O K 


1 ^ 
1 ~ 


1 9 


N t 2 2 2 A 


1 o 


1 9 


N ' 2 2 2 A 


1 7 
1 f 




N .- 3 t; 5 C 


i 




N ' 3 5 C 


n 
e 


1 


M 3 C 5 C 


t 


o 

c. 


N 3 ".5C 




i 
1 


fj 3 ,'5C 


fi 


il 


N 3 0 5 C 


6 


1 


N * 3 C 5C 


7 


2 


. 3 C 5 C 


8 


o 

£ 


H 3 0 5 C 


9 


J 7 


M . ■ 3 J 5 C 


1 0 


1 


N. 3 C*5C 


1 1 


2 


N ^3 ^5 C 


1 2 


d 


N ' 3 ISC 


1 3 


1 


ti 3 0 b C 


. I 4 


9 


N .3C5C 


1 5 


1 2 


N ' 3 'i 5 C 


1 C 


1 2 


N * 3 0 5 C 


I 7 


6 


T 1 0 5 A 


1 




T ; 10*3 A 


2 


1 


T C ^) A 


, 3 


£ 


T T 1 0 5 A 




1 
1 


T '^1 1 C 5 A 


5 


2 


T M 1 b A 


6 


1 


T 1 ' 5 A 


7 


d 


; T 1 C5A 


8 


2 


T ';1C5A 


9 


17 




1 0 


1 


T aC5A 


1 1 


2 


Ti 105A 


1 2 


2 


TjIOSA 


13 


1 




1 4 


9 


T V U 5 A 


1 5 


12 


TD1C5A 


16 


12 


TR1C5A 


17 


6 



ERIC 



CHI SQUAREDS SiGNlFtCANCE LEVELS 



nr c T nw 

U L O 1 \3v% 






DESIGN 






6 A SLD 


SRS ADJUSTED 


BASED 


SRS 


ADJUSTED 


•15,78 


7.50 


5.98 


.015? 


• 2 769 


• 4256 


C • 27 


0.81 


0.65 


• 6 02 2 


' «»74 


. 42 1 w 


U • k'O 


1 .46 


1.16 


.986 9 


• 4 826 


. 55 95 


4 • 76 


4.13 


3.29 


. 029 1 


• 0422 


. 0697 




4.14 


3 . 3 C> 


. 84 71 


,12 63 


.19 22 


It • 3 i 


2.13 


1.69 


.4732 


• 1 4 4 8 


.1931 


L • i O 


0.80 


0.64 


.9134 


• 67 0 0 


. 72 63 


0 • 34 


3.99 


3.18 


• B4 2 1 


• 1 358 


.2037 


2 72 • 17 


30.69 


24.46 


. 0 J n 0 


• 'J 2 1 8 


. 1 075 


• 0 1 


C.02 


0.01 


. 9398 


• B991 


. 90 99 




2.76 


2.20 


.2310 


• 251 3 


. 332b 


15*62 


1C.29 


8.20 


. 00 04 


• 0{i58 


• 0 1 6 5 


1 0 • 92 


5."25 


4.18 


.1010 


• C 220 


• 04 09 


3 7 • 64 


13.70 . 


1C.92 


. C 0 U 0 


• 1*3 34 


• 2 8 1 4 




1 7. 06 


-13.60 


• 00 uC 


• 1 4 72 


• 32 71 


2 4 • 4 9 


1 7.69 


14.09 


' .0180 


• 1 255 


. 2947 


1 c 70 


7.5'J 


5.98 


.0150 


• 2 769 


• 42 56 


14*50 


2 2. C5 


16.28 


. 0245 


• 0012 


.0123 


L • l« i 


1.16 


0;B5 


.9378 


• 2 82 0 


• 3 5 5 3 


(k 9 Q 


' 0.55 


0.4 1 


.0710 


• 75 79 


.8149 


i5 • 3 9 


5.C6 


3.73 


n <. s 


• 0 2 45 


• 0533 


11^ 9 Q 


1 .73 


1.27 


. C U 0 5 


• 4 2 1 8 


• 52 88 


^ • uO 


1.18 


0.87 


.8722 


• 2 775 


. 55 


3*69 


1 .02 


0.76 


.15 81 


• 5 9 95 


■ .6854 


2 • 1 1 


1.35 


l.CO 


.3663 


- s f 1 ft q 

• 3 w 0 7 


• 6073 


24 8* C 8 


4 7.01 


34.70 


n f. fi 'v 

. U U U 


1 1 

• J J U i 


• fi C 6 8 


f • 49 


0.59 


0.44 


. 4 ft 4 7 


• 4 4 2 2 


• 5 0 9 1 


r • 44 


0.81 


0.59 


.8^10 


• 6 6 8 6 


• 74 2 9 


7 Wi 
1 tHd 


• 7.19 


5.31 


^ . C2 4 5 


. i> 2 75 


• 07 C 4 


L • 1 1 ' 


0.13 


0.10 


.7366 


• 7 1 6 7 


• 7 5 5 3 


2 D • 74 


6.70 


4.95 


fl 1 M 
. U i 0 7 


• 6 6 8 1 


M ^ ft ^ 
• a 0 0 7 


1 U Ti • 1 1 . 


23*08 


17.04 


.0000 


• 0 2 7 0 


• 1 4 8 1 


£ £ 9 7 
fa P • £ / 


38.15 


28.16 


. 0 if c c 


• 0001 


.0052 


I 4 • Ou 


2 2. U5 


16.28 


0 . 0 2 4 5 


ft ft 1 9 
• U U i £ 


.0123 




33.11 


2 0.55 


'1 f n ft 
• J L u u 


n fl n (1 

• U U V u 


• 00 2 2 


1 5 • ? 4 


1.44 


0.69 


. u u n 1 


0 x n y 

• £ 0 'J i 




5 • 83 


5.92 


3.68 


n K A T 


• |j 3 A f 


* * -J y J 




55.83 


34.66 


fl ft ft ! 


• 0 0 0 0 


• 0 0 C 3 


(1 • 85 


5.91 


3.67 


£ S 4 A 
. D U t D 


• 352 1 


• A 3 7 I 


14,27 


0.52 • • 


0.32 


. 0302 


• 4712 


• 5703 


10.69 


1.48 


0.92 


.0048 


• 4779 


• 6324 


5. 02 


' 6.63 


4.11 


.0812 


• 0363 


• 1278 


n03,92 


156.36 


97.06 


. 0300 


^0000 


• 0000 


2,93 


4.24 


2.63 


. 0867 


• 0394 


• 1047 


2.84 


9.19 


5.70 


.2413 


• ClOl 


• 05 78 


49.74 


67.02 


4 1.60 


. 0000 


• 0 000 


• 0000 


06 


n.io 


0.06 


• 8056 


• 7547 


• 8056 


234.58 


30.39 


18.87 


..0000 


• U0U4 


• 0264 


117.33 


4 5.94 


26.52 


.0000- 


• cooa 


' •0046 


354. 06 


124.54 ' 


77.30 


• 0000 


• 0000 


• 0000 


190.59 


33.11 


20.55 


.0000 


• 0000 


• 0022 
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Table B-2, (continued) 



CHI SQUAREOS 



SIGNIFICANCE LEVELS 




i T L .M > 


ttST 




DESIGN 


SRS 




u DESIGN 






NUMBER 


NUMBER 


D.F. 

_^ 


BASED 


AOJUSTED^ 


BASED 


SRS 


ADJUSTE 


t;ii3a 


1 


6 


^ . 19 


2 . 39 


fl 

1,60 


.6514 


.9112 


. 9529 


T • 1 1 0 A 


2 


1 


U. 13 


C.21 


0.16 , 


.7135 


.6457 


,6079 


f •* 1 ; 0 ^ 


3 ; 


2 


16.66 


16.14 


12.32 


. (1002 


.Ou J3 


..0021- 


T '.IK A 




1 


25.66 


1 7,5a 


13,36 


. 0000 


, 0000 


^003 


T 1 1 C A 


5 


^ 2 , 


7.36 


2,73 


2.09 


. 02^52 


. 254'8 


. J521 


T 1 1 0 A 


6 


1 


2 .^7 


2.58 


1.97 


. 14 64 


.1081 


.1604 


T ,11'!A 


7v 


2 


5.57 


4.68 


3,57 


.0617 


.0962 


.1674 


T'UOA 


8 


2 


7.57^ 


' 3.35 


2.56 


. 022 7 


> 1 H 6 8 


.2779 


t^ii:a 


9 


17 


445. 


82 .67 


63.11 


, 0000 


^ , 0000 


• uooo 


T'il TA 


1 0 


i 


1 4 . 2 1 


5.14 


3,9 3 


.0002 , 


,uJ33 


. 04 75 


T • 1 1 A 


1 1 


2 


31 . 12 


3 1 , 90 


24,35 


,0003 


• COOO 


.0000^ 


til :^^ 


1 2 


2 ' 


27.58 


18.15 


' 13,85 


. DC uO 


. 0 0 0 1 


,0010 


T M 1 jA 


1 3 


1 


0. 03 


0. 11 


0,08 


.8595 


,7439 


.7753 


T * I 1 (J A 


1 ^ 


9 


35.32 


• 19,42 


•14,82 


. 0001 


,0219 


. 0959 


T 11 * A 


15 


12 


2 83 .60 


41 .33 


31.55 


,0000 


..OOCii 


. 0016 


T U 3 A 


1 6 


12 


4C.20 


2 6.93 


20.54 


. QGOl 


• G080 


. 0576 


T IICA 


17 


S 


4 . 19 


2,09 


1.'6 0 


.6514 


.9112 


.9529 


T 21.3A 


1 


6 


1 1 . 78 


5.73 


3,5 3 


, 0671 


,4537 


.7396 


T 2«,'3A 


2 


1 


2.69 


1 ,03 


0.64 


.1012 . 


.3092 


.4248 


T 2C3A 


^ 3 


2 


12.96 


5.19 


o 3.aa 


.rD15 ^ 


. 0746 


. 2021 


T ''2 t'3A ' 




1 


56. 15 


5 1.33 


31,62 


.0000 


. 0 00'^ 


.0003 


T -.21 3 A 


5 


2 


r.35 


.1.37 


0,85 


.8397 


.5033 


. .6551 


T )2C3A 


6 


1 


* . 4 3 


0.P6 


0.04 


,5143 


.8088 


. 84 9? 


T'*2C3A 


7 


2 


7i2ft 


5,69 


3,50 


. 0262 


. 0582 


.1735 


T ..'2 0 3 A 


B 


2 




e. 12 


C,37 


.76D3 


.9417 


. 9637 


T'.2^:3A 




17 


525.61 


142,34 


87.70 


,0000 


.uCCU 


. 0 L' 0 0 


T'2u3 A 


1 U 


1 


5.4<» 


6 ,U0 


3.7C 


,0197 


,C143 


.0545 


T *2C3A 


1 1 


2 


0.39 


0.73 


0.45 


,8210 


.6957 


,7997 


TI-203A 


1 2 


2 


39.69 


5 0, 15 


30.90 


,0000 


. 0030 


. 3000 


T'^2 03A 


1 3 


1 


1.23 


C.31 


0.19 


.6339 


.5754 


1 .66C2 


T'^2u3A 


1 4 


9 


38.83 


4 1 ,92 


25,83 


'.cooo 


,0000 


. 0022 


T:2n3A 


t 5 


12 


55.45 


18,35 


11.30 


. 0000 


.1055 


.5030 


T '^CSA 


1 6 


12 ' 


147.27 


91,34 


56.28 


. 0000 


,O0OC 


. OUOG 


T i2r.3A 


i 7 


6 


1 1 . 78 


, 5.73 


3.5 3 


. 0671 


.4537 


. 7396 


T 2a3A 


1 « 


6 


24 .96 


13 .69 


10.92 


^.0003 


.0333 


. 0908 


T 2?.3A 


2 


1 


1 . 86 


2,50 


2,00 


,1724 


.1136 


. 1576 


T '223A 


3 


2 


l'.96 


3.61 


2.88 


.6179 


,1641 


. 2365 


T •223A 




1 


5.48 


6,52 


5. 20 


ar9 2 


' ,0107 


. 0226 


Tr223A 


5 ' 


/ 2 


2.05 


0,46 


0.36 


.'3j5 97 


,7961 


, 8337 


t:223a 


6 


1 


r .41 


4.27 


3.41 


,5T^8 


.(1368 


. 0650 


T '»223A 


7 


2 


£.67 


3.37 


2.69 


. 26 34 


,1856 


• 2609 


T 223A 


H 


2 


3.59 


2 ,04 


1.63 


,1661 


,3601 


,4428 


T».??3A 


9 


17 


106 .90 


53 ,68 


42,82 


• 0 0 0 0 


,3000 


, 0005 


T. 2 23 A 


1 0 




C. 05 


(i , 04 


0,03 


. 8244 


.83 44 


.8519 


t:'223a 


11 


2 


1 .53 


1.^9 


1,11 


.4652 


.'*990 


.574'* 


T 5223A 


1 2 


2 


12.^0 


1 0,71 


8.54 


.0320 


.0047 


. 0140 


T .'^23 A 


1 3 


1 


0.83 


0,77 


0.62 


-3611 . 


.3788 


.4318 


T' 223A 


1 ^ 




2 2.55 


16,21 


12.93 


• 0073 


,0626 


.1657 


T 7223A 


15 


12 


47,94 


21,63 


17.25 


, 0000 


,3419 


. 1403 


T '•223A 


16 


12 


37,83 


4 2 ,60 


33.98 


. 0002 


• 0000 


,0007 


T.-.223A 


17 


6 


2 4,96 


13,69 


10.92 


. 0003 


.0333 


• 09C8 
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Table B-2, (continued) 
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CHI SQUAREOS ' SIGNIFICANCE LEVELS 
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2 
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tS * 3 O 
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* u u u u 
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* VA O r 






1 
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7 '7 T 7 


A Q 'to 
D O * J V 


* J'AMV 
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* 0 3 C 0 


S . 1 C' 8 A 


7) 


c 


-n A 1 


0^17 
U * i r 


U * i 3 


* 0 1 4 9 


^ q 1 qA 

* 7 A 7 ^ ; 


* 92 87 


o 1 v c A 


D 


1 

• 1 


£ • 3 £ 


U * 1 J 


fl - 1 7 
U * 1 r 


* 1 2 74 


^ £ £ ft 1 
* D D U 1 


* 6799 


C * 1 U A 
r> 1 ij 0 A 




9 


0 'i • £ £ 


Jl A * 7"f 


1 0 - S 1 


. , 0 n D 0 


* 0 0 2 6 


* U si 3 £ 


<i ** 1 O ft A 
o 1 V C « 


a 

CI 


o 


4 • 91 


w * 1 D 


0*14 


* 1^ 8 5 9 . 


* 92 2 6 


* 93 1 5 


^ * \ 1 1 ft A 


- 9 


1 7 


1 r AS . 9 A 


1 1 2 * 83 


9 9 * 3 U 


* I? 0 1 0 


* 0 0 0 D 


,1)0 00 


S 1 C 8 A 


1 J 


J 


3 • OD 


1 * C8 


0*95 


* 0B32 


* 29 93 


* 33 02, 


c 't 1 rat 


1 A 


2 




7 * 04 


6*23 


,26 09 


* 02 96 


* G452 


<; " J g ^ 


1 2 


2 


56*34 


11 • 5fa 


68*26 


• c too 


* 0 0 00 


* oocn 


S 1 ^ 8 A 


1 3 


I 


2 • 24 


1*43 


1*26 


* 1344 


* 23 17 


* 26 19 


S ii 1 ' 8 A 




q 


2 3*19 


8*14 


7*16 


* 0 05 8 


*52 04 


* 62 


f: ' 1 C 8 A 


1 5 


1 2 


111*57 


3 3*51 


29*4 9 


* 00 0 0 


* 00 08 


* D033 


S ' 1 8 A c 


1 6 


1 2 


1 93 * 05 


9 8*54 


86*73 


* G 0 0 0 


* 0 0 u 0 


*oooc 


S * 1 C % A 


1 7 


6 


4 1 * 3B 


1 3 * 94 


12*27 


* 000 0 


* 03 0 3 


* 0562 


o • 1 i i A 


1 
1 




18*47 


2 8*19 


24*72 


005 2 


* J 0 0 1 


* 00 04 


C ' 1 1 7 A 
o 1 1 r A 


o 


1 


1 '.1*77 


D * O A 


5*53 


* CO 1 J 


- P 1 9 '1 
* l> A £ J 


* 31 87 


o 1 1 r A 




£ 


A - 9S 
C * c3 


3*83 


J * J D 


. n 1 A 1 

* U A D A 


^ 1 A 7 Ti 
* A t r U 


* 1 8 6 2 


S I 1 7 A 




1 
i 


3 7* * i r 


9 9 7A 
£ £ * r *t 


1 Q Q A 
1 7 * 7 


- n n 1 n 

* U U u u 


- ft n rt 

* J u u u 


* il U U J 


SOI I 7 A 




o 
£ 


1 D * 3 1 


11 A A 


in n X 

1 u * U 3 


* U U U J 


* U w J J 


ft ft A A 
* U U b 9 


S ' 1 1 7 A 


6 


1* 


0 rt9 
£ * 'y£ 


C £ £ 
* DD 


A a £ 

^ * 7 O 


1 R R 1 
* 1 3 3 i 


iT 1 7 A 
* I' 1 f ^ 


0 9 A f. 
* ^£ bu 


S ' 1 1 T A 


7 


<r 


£ * 


9 R 3 
£ * 3c 


9 9 1 
£ * £ 1 


T £ 1 Q 
* J D i 7 


9 fi T fi 
* £ O 3 O 


* 3 3 1 3 


S * 1 1 7 A 


B 


2 


5*42 


1 D * 1 9 


8*92 


£ £ R 

* Jb D 3 


f 1 il £ 9 

* U Ub £ 


fli 1 
* vj 1 1 3 


S * 1 1 7A 


9 


1 7 


2 562*63 


7 2 * 84 


63*87 


n rt /t A 
* Q U 0 0 


^ A n ."1 
* 11 U u u 


n 11 n A 
* UUGO 


S 1 1 7 A 




1 


7*21 


4*41 


3*86 


* 0 j 73 


'"l T C« fi 

* J J 3 H 


- ^ * 04 93 


S 1 1 7 A 


11 


2 


7 * 


T 17 
0 * 1 r 


9 7 Q 

£ * r o 


rt Tj A Q 
* U W b 7 


9 m 1 

* £ J 3 i 


9 A qx 

• £ 7 73 


S * 1 1 7 A 


1 2 


2 


4 8*63 


9 7 £ Q 
£ r * DO 


9 A 9 7 

£ *i * £ r 


fi rt A »\ 

* U U U u 


n ft Ti ft" 
* J u u u 


fl ft ft 


S * 1 1 7 A 


1 3 


1 


(■ * V £ 


3 * 05 


U *'w 3 


u a A 9 
* r o ^ £ 


* 3 1 9 4 


Q T A 7 

* H J U r 


S * 1 I 7 A 




9 


4 8*46 


12*29 


10*78 


* (« U U U 


1 a 7 A 

* 1 7 f 


* 29 1 4 


b i i r A 


D 




9 f S 1 (^ 9 
£ 1 * 1. £ 


J O * r O 


J X * J 1 


* J ^ W V 


^ n A ft A 
* u u u ^ 


ft rk 1 7 

* U J 1 r 


S ' 1 1 7 A 


1 D 




1 7 Q 17 
1 • 7 * i, r 


A A Q n 
O *f * 7 U 


3D * 7 U 


f. iT fi ft 
* U u U U 


n A ft 0 

* U U U J 


n ft ft ft 
* u u u u 


Si 1 I 7 a 


1 ' 


£ 

b 


1 U ^7 
J ^ i. O * ^ 1 


9 Q 1 Q 
£ O * 1 7 


9 A 7 9 
£ 1 * r £ 


n 9 

* U U 3 £ 


* U U U 1 


* u u 


S ' 2 U6 A 


1 


6 


7*72 


9*81 


3 * H 7 


9 R a 

* £ 3 7 J 


1 T T 1% 

* 1 J J u 


A T C« £ 
* *l J 3 9 


S V 2 0 6 A 


2 


1 


9 7 A A 
£ r * 




9 9 T 
£ * £ .3 


A fk A n 

* U U u U 


n C« A 9 
* J 3 ^ £ 


1 •\7 
* A 3 3< 1 


S * 2 0 6 A 


3 


2 


9 9 Q 
£ * £ 7 


1 Q A 
1 * 7*1 


1 1 A 

1 * 1 b 


X 1 fi A 


T 7 q 7 
* J r 7 r 


^ ^ q n 

* 3 3 7 U 


S 2 06 A 




1 


9 t A 11 
£ 1 H * 1 1 


Q Q U 7 

o o *.b r 


'3 J * J o 


n h n 
* U y U U 


n n n n 
* U U U U 


<i n A '\ 


S r 2 G6 A 




2 


T 9 A 1 
3 £ * H 1 


1 l« 9 A 
1 3 * £*l 


9*15 


'» ft A 
* J U U u 


.'I ft A R 
* ll U U3 


ft 1 r x 
* U 1 uO 


S ' 2 !• 6 A 




1 


.V 1 D * 7 J 


1 ^ fl(^ 

1 * nri 


1 11 

1*11 


* u ki 


1 7 ^ 
* i r w J 


^ 9 q 1 « 

* £ 7 A J 


^ ' 0 f* C A 

o • c u n A 


7 


2 


4 * 05 


2*45 


1*47 


* 1 3^ 0 


* 2t9 3 1 


* 4 7 85 


S' 206A 


8 


2 


27*79 


13*26 


7*96 


*0UU3 


*b.0l3 


* 0187 


S :2^6A 


9 


17 


2696* ('3 


293*69 


176*40 


' * 0000 


*COUO 


*OOGe 


s'^r'^.f.A 


1 r> 


1 


8*36 


5*38 


3*23 


*nC38 


*0203 


*0721 


S '2L«6A 


1 1 


2 


C*30 


9*35 


0*^1 


*B612 


*83B5 


'*89.9S' 


S:20SA 


12 


2 


127*86 


83*99 


50*45 


* CiOOO 


* 0000 


* opou 


S - 2 06 A 


1 3 


1 


0*38 


0*34 


0 * 2 0' 


*5384 


*5592 


*65i*3 


S'-2CS A 


M 


9 


178*38 


51*24 


30*78 


*nooo 


*0000 


*0003> 


s :'2:»6A 


15 


12 


361*28 


37.89 


22*76 


*QOon 


*00G2 


*0298 


SC2rf,A 


16 


12 


1763*83 


240*56 


1.4 4*4 9 


.* 0000 


*OC00 


*oooa 


St>2C6A 


1 7 


6 


7*72 


9*81 


5*89 


*2590 


*1330 


• 4356 



(continued) 



NAEP 
.ITETW TrST 
MUMBCR NUMBER D.F, 



S'225A 1 f. 

r>'225A 2 1 

S 225A 3 2 

S-22bA ^ 1 

S'225A 5 2 

s:225A 6 1 

S 2 25.A 7 2 

Sr225A 8 ? 

S 225A 9 17 

S 225 A 1 r» • 1 

S''225A 11 2 

S-225A 12 2 

S'225A. 13 1 

S 225A 14 9 

S'225A 15 12 

S^225A If. 12 

S 225A 17 6 



CHI SQUAREDS 

DESIGN 

BASED SRS ADJUSTED 



7 • 7C» 


7.20 


6.90 


''.•20 


0.60 


0.57 


.16 . 09 


1.8<( 


1.77 


95. 10 


'^0.18 


46.18 


1.22 


1.52 


1.46 


6.75 


3.78 


3.62 


2L.62 


1.G6 


1.G2 


i.n9 


1.55 




592.72 


133.88 


128.31 


19. '^6 


6.86 


6.57 


1.78 


2.55 


y.^5 


57. 13 


47.96 


45.97 


1.61 


0.66 


0.63 


59.05 


27. b5 


26.41 


24.9<» 


6.26 


6.03 


233. 7<l 


105.97 


101.56 


7.73 


7.20 


6.90 



, SIGNIFICANCE LEVELS... 
DESIGN 

BASED SRS ADJUSTED 



.2611 


.3031 


.3305 


.6544 


.'»397 


• 4493 


. 0003 


.3979 


.4134 


.ncoo . 


. 9000 


• 0000 


.54 35 


.<^672 


.4823 


.0094 


.0518 . 


• CI570 


• uOOO 


.5883 


.6014 


.5790 


.<^606 


.4757 


.OQOC 


.0000 


.OQCO 


• 0000 


• 0088 


• niC4 




.2789 


»2941 


.0000 


.OUOO 


.0000 


• 2049 


.4161 


.4260 


• 0900 


.0011 


• 0018 


.C151 


.9022 


.9160 


. 0000 


.OOUO 


• 0000 


.2611 


.3031 


.3305 
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Table Mean Score Wald Statistic Chi Squareds fof the Race*Sex* PARED 

Cross'-Classlflcatlon 



CHI SQUAREDS 



SIGNIFICANCE LEVELS 



NAEP 



ITEM 


TEST 




DESIGN 




NUMBER 


NUMBER 


D.F. 


BASED 


SRS 


AGE '9 


1 




54. 14 


15.88 


AGE .9 


2 




0.93 


3.47 


AGE .9 





1__ 


1LU7_ 


1.09 


ACE 9 




1 


18.63 


3 5 . 4 0 


A&E.'9 
ACE v9 


5 


1 


2.59 


0.55 


6 


1 


4.59 


1.03 


AGE''9 


7 


1 


5.68 


0.63 


AGE :9 


8 


1 


2. 79 


0.57 


AG.t:9 . 


9 


11 


2C;4.52 


165.65 


AGE >9 


■ 1 0 


1 


28.64 


47.86 


ACE''9 


1 1 


1 


1 ;iG 


1 .50 


age:-9 


1 2 


2 


32.26 


48.17 


AGE . 9 


1 3 


' 1 


5.22 


4.74 


AGE 9 


1 4 


6 


76.10 


74.36 


AGE ..9 


1 5 


6 


5. '.9 


3.86 


AGlC,9 


16 


8 


83.20 


6 3.10 


AGt *9 


1 7 




54.14 


15.88 


AGE 13 


1 


4 


1.74 


3.19 


ACEi3 


2 


1 


59.33 


2 4.52 


AGLi3 . 


3 


1 


a. 16 


0.02 


.AGE13 


■ 4 - 


1 


97. 32 


103. 15 


AGE 13 


5 


1 


P. 17 


0.58 


' AGE 13 


6 


1 




0.01 


AGE13 


7 


1 


(>.(}3 


0.21 


AGE13 


8 


1 


0.02 


0.03 


AGE : 3 


9 


11 


6r)9.23 


462.33 


AGE13 


1 ri 


1 


75.^43 


168.38 


AGE 13 


1 1 


1 


1.24 


1 .48 


AGE13 


1 2 


2 


6 1.34 


10 4.64 


AGE;13 


1 3 


1 


C.20 


U.28 


AGE13 


14 


6 


159.30 


180.27 


AGE13 


1 5 


6 


31.93 


18.71 


AGE13 


16 


8 


226.51 


151.74 


ACE13 


1 7 


4 


1.74 


3.19 


AGE17 


1 


A 


2.80 


3.22 


" AGE17 


2 


1 


57. 01 


2 4.96 


AGE 1 7 


3 


1 


2.44 


2. 83 


AGE17 






81.54 


82.43 


AGE17 


5 


1 


ft.'J9 


r.oo 


AGE17 


6 


1 


4.68 


1.96 


AGE17 


7 


1 


0. 06 


U.OO 


AGE17 


8 




0 . 4'9 


0.14 


. ACE 17 


9 


11 


1735.18 


735.85 


AGE 1 7 


1 0 




211.20 


261.44 


AGE17 


1 1 


1 


24.29 


19.16 


AGEl 7 


1 2 


2 


74.93 


80.89 


A GE I 7 


13 


1 


2.27 


2.44 


AGE 17 


M 


6 


223.27 


294.93 


AGE17 


15 


6 


55.23 


26.99 


AGE17 


16 


6 


356.80 


142.30 


AGE17 


17 


4 


' 2.80 


3.22 


o 










ERLC 





















ADJUSTED 

1C.59 
2.31 
0.72 

2 3.62 
0.36 
0.69 
0.42 

0. 38 
110.52 

31.93 

1. uO 
32.14 

3.16 
49.61 
.2.57 
. 42.10 
10.59 
2.18 
16.81 
0.01 
70.70 
0.40 
0.01 
0.15 
0.02 
316.88 
115.41 
1.01 
71.72 
0.19 
123.56 
12.82 
104. 00 
2.18 
2.77 
21.45 
2.44 
70.84 
0.00 
1.69 
Q.OU 
0.12 
629.82 
224.68 
- 16.47 
69.52 
2.10 
253.47 
23.20 
122.30 
2.77 



DES IGN 
BASED 

.DOOC 
.3353 
.0,074 
. OCOO 
.1073 
.032 1 
.0153 
.0=946 
.0000 
.0000 
.2938 
. 0000 
.0223 
. . 0000 
.5324 
. 0000 

• 3CC0 
. 7827 
.OiiOO 

• 6899 
.QjGO 
.6810 
.9990 
.8694 
.8945 

.nooo 
.noco 

.2651 
.0000 

• 6539 

.cuoo 

. OOO'O 
. 0300 
. 7827 
.5924 
.0000 
.1181 
.U!)DO 
.7661 
. 0306 
.8144 
.4840 
.(fSOO 
. 0000 
. 0000 
.0000 
.1319 

• uooo 

. CO 00 

.o&oo 

• 5924 



SRS 

.0032 
.0626 

• 2976 
.0000 
.4597 
.31C2 
.4275 
.4498 
. 0000 
.OOCO 
.2209 
.0000 
.0295 
.0000 
.6962 
.OCOO 
.0032 
.5273 
.0000 

• 9023 
.0000 
.4447 
.9184 
.6437 
.9556 
.0000 
.0000 

• 2239 
.0000 
.5992 
.0000 
.0047 
. 0000 
.5273 
.52.15 
.0000 
.0923 
. 0000 
.9641 

• 1610 
.9829 
.7096 
.0000 
.0000 
. 0000 

• 0000 
.1180 
.OUOO 
.0001/ 
.0000 

• 5215 



ADJUSTED 

. Q315 
.1282 
.3949 
.OOCO 
.5459 
.4072 
.5169 
.53 71 
. 0000 
. 0000 
.3174 
.OOCO 
. 07 55 

. 0 0 on 

.8603 
. 0000 
. 0315 
. 7021 
_ .0200 
.9191 
.OQOO 
.5269 
.9324 
.7018 
.88 03 
. COOO 
.0000 
.3143 
. 0000 
.6635 
. u 0 0 0 
. 0459 
. 0000 
. 7021 
.5973 
. 0000 
.1186 
.'0003 
.9852 
.1939 

• 9B41 
. 72 99 
. 0001) 

. oooc 

. 0000 

• 0000 
.1473 
. 0000 
. 0007 

• 0000 

• 5973 
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Table Mean Score Uald. Statistic -ehl"^5^^^^^ the Sex*TOC*PARED 

'Cross-Classification 



CHI SQUAREOS SIGNIFKANCE LEVELS 



M A E P 


— 


- 






a 








ITEM 


T r C T 










DESIGN 






Hi 1 1 M n C* D 


IMUn H L n 


n c 


D A O L u 


SRS 


ADJUSTED 


BA SED 


S R S 


ADJUSTED 


AGE J9 


X 


o 


2 2*53 


1 4 * 36 


8*67 


* 001 0 


*0259 


*1930 


AGE ? 9 


o 


t 


7*75 


£ *04 


1*23 


* 0C54 


* 1531 


*267G 


A G E 9 


o 


C 


* i o 


r * 33 


4*56 


* 01 02 


* 0230 


*1 024 


. AGE w-9 _ 




1 


■« 7 ^ A n 

J f * *T II 


• 5 2*09 


31*45 


* G 0 0 0 


* 0000 


* 0000 


A r r ■ Q 


R 


c 


11*46 


2*65 


1*6 3 


*0032 


*2658 


*4493 


A U L " 


O 


1 


13*51 


3*49 


2*11 


*0C02 


* 0 6 1 9 


*1468 


A G E ^ 9 


7 


£ 


12*04 


6*33 


3*82 


* 0024 


•.0422 


* 1 4 8 0' 


A GE * 9 


O 




2 2*46 


5 •93^ 


3*58 


* 000 0 


• 0515 


*1668 


AGE C 9 


q 


1 r 


274 * 45 


1 1 8 * 98 


71 *84 


* COO 0 


*0000 


*oooa 


AGE !) 9 


\ ft 




r * Gi 


0*31 


0 *Q 1 


*93 08 


*9252 


*941S 


AGE 9 


1-1 

« i 1 




"1 * 57 


3 * 96 


2*39 


*4569 


* 1 379 


*3a23 


AGE C 9 




0 


4 2 * 96 


5 7*90 


34*96 


* 00 0 0 


*3000 


*U000 


AGE ''9 


1 3 


t 
1 


10*69 


6*57 


3*96 


* DOM 


*01 04 


* 0465 


A GE ' 9 


\ ft 
L "f 


q 


2 2*44 


13*76 


8*31 


* 03 76 


*1310 


*5932 


AOL ~ ^ 


I 3 


1 9 

1 £ 


X £ O * *f O 


2 4*93 


15*05 


* 0 0 0 Q 


* 0152 


*238S 


AGE ^9 


1 A 


1 2 


1 7 3 * 84 


10 4*41 


63* 04 


* 0000 


«oaoo 


* 00 (>3 


A GE j9 


1 1 




2 2*53 


14*36 


8*6 7 


* 00 1 0 


*0259 ' 


* 1933 


AGE 1 3 


-1 « 
1 


O 


2 5 * 92 


• 13*91 


8*38 


* 00 02 


*0306 


*2116 


AGE 1 3 




1 
1 


3 * 87 


D * 1 9 


0*12 


* 04 91 


*6609 


* 7335 


AGE 1 3 




c 


39*2? 


12*29 


7*4 0 


* CO 0 0 


* 002 1 


*U24/ 


AGE 1 3 




1 

1 


9<« Q > flQ 


13 9*61 


8 4*09 


* fl 0 0 0 


, 0000 


* 0000 


AGE 1 3 




o 

c. 


1 * 54 


3*31 


1*81 


* 46 33 


*22 1 9 


*4038 


AGE 1 3 


b 


1 
1 


1*21 


1*59 


0*96 


* 2 7 2 0 


5 * 2 067 0 


* 3271 


AGE 1 3 


7 
f 




39* 49 


6*28 


3*78 


* 00 0 0 


* 04 33 


* 15 1 0 


A GE 13 


6 


£ 


19 

^ * 1 £ 


3*46 


2*08 


,2104 


* 1 774 


*3528 


AGE 1 3 


9 


\ 7 
1 r 


1 6 8 6 * C 4 


2 7 4 * 84 


165 *55 


* 0 0 0 f) 


*booo 


*000& 


AGE 1 3 


1 0 


i 




1*72 


1*03 


* 1 2 2 2 


; 1 903 


* 3094 


AGE 1 3 


1 1 


£ 


D * OO 


16*12 


9*71 


* 05 2 8 


* 0003 


* 0378 


AGE 1 3 


I c 




1 S Q * 9A 
1 □ 7 * £0 


13 5*15 


81*41 


,0000 


,0000 


.* 00 00 


AGE 1 3 


1 T 
i 0 


1 


C * 01 


0*03 


0*02 


*9l 35 


*854C 


*8864 


AGE 13 


I ^ 


q 


2 9*59 


2 6*23 


15*80 


* 00 05 


* 001 9 


* 0712 


AGE 1 3 




1 £ 


66*24 


3 4*03 


2 0*5 0 


* 000 0 


* 0007 


* 0563 


AGE 1 3 


1 O 


1 9 
i £ 


451*30 


215*98 


13 0*10 


* 0000 


* 0000 


* 0000 


AGE 1 3 


« 7 


c 


2 5 * 92 


13*91 


8*38 


,00 0? 


* 03 06 


*21 16 


AGE 1 7 


1 


t 

o 


9*01 


6*23 


4*29 


,17 32 


*3977 


*6373 


^GE 1 7 


2 


^ . 1 


1 * 84 


C * 03 


0*02 


* 1 75 3 


* 8 6 4 & 


*8874 


AGE 1 7 


3 


£ 


7*30 


4*44' 


3*05 


* U 2 6 0 


* 1 089 


*21 72 


AGE 1 7 


^ 


1 


947*62 


18 2*14 


125*43' 


, 0000 


* 0000 


* 00 00 


AGE J 7 


5 


£ 


11* 


1 C * 35 


7*13 


,0001 


* 0 056 


* 0283 


AdT 1 7 


6 


1 


U * U 1 


1 9 9 
1 * £ £ 


0*84 


* 9 4 Irl 


* 2696 


*3596 


AG E 1 7 


7 


2 


J * u 


^ A7 
O * O 1 


2*66 


• 2088 


* 1 445 


* 26 39 


AGE 1 7 


B 


2 


5. 15 


7*40 


5*C9 


" *0763 


* 0248 


* 0783 


AGE 17 


9 


17 


117C8*6 


350*66 


241*49 


, 0000 


*0000 


*doao 


AGE17 


10 


1 


10*61 


5*34 


3*68 


*C011 


*D209 


*0552 


AGEI7 


1 1 


2 


2*17 


2*75 


1*89 


*3379 


*2534 


*3885 


AGl17 


12 


2 


276*08 


181 *79 


125*19 


*0C00 


*0000 


*O00D 


AGE 17 


1 3 


1 


C*05 


0*04 


0*03 


*8314 


*8437 


*870a 


AGE17 




9 


129*18 


43*21 


29*76 


*dooo 


*ouoo 


" *C005 


AGE17 


15 


12 


112*17 


2 3*05 


15*87 


*cooo 


* 0273 


*1971 


AGE 17 


16 


12 


1389*07 


305*93 


210*68 


* 0000 


* 0000 


« 0000 


ACE17 


17 


6 


9*01 


6*23 


4*29 


*1732 


*3977 


*6373 
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Table C-1. Balanced Effect F-Tests for 9-Year-Olds 



Design Based 
d.f ■ F Prob 

(30 denominator d.f.) 



Unweighted 



Weighted 
F Prob 



Prob 

(2457 denominator d.f.) 



N0222A 
Sex 
Race 
PARED 



0.77 
.7.76 
4.84 



.39 
.01 
.01 



0.98 
12.66 
1.64 



.32 
.00 
. 18 



0.82 
5.33 
2.76 



.37 
.02 
.04 



N0227A 
Sex 
Race 
PARED 



1 
1 

--3- 



2.21 
15. 19 
39.76 



.15 
.00 
.00 



1.05 
22.27 
15.75 



.31 
.00 
.00 



1.85 
14.64 
17.94 



. 17 
.00 
,00 



N0222A 
Sex 
TOC 
PARED 



1 
2 
3 



0.89 
0.66 
6.27 



.35 
.52. 
.00 



1.10 
0.30 
2.64 



.29 
.74 
.05 



0.95 
0.54 
3.62 



.33 
.58 
.01 



N0227A 
Sex 
TOC 
PARED 



1 
2 

3 



1.95 
2.76 
56.72 



.17 
.08 

,00 



0.98 
2.89 
19.25 



.32 
.06 

,00 



1.64 
3.23 
20.81 



.20 
,04 
,00 



N0305C 
Sex 
Race 
PARED 



1.79 
77.32 
10.60 



. 19 
.00 
,00 



0.85 
126.34 
10.69 



.36 
.00 
.00 



1.99 
126,25 
14.83 



.16 
.00 
.00 



N0317A 
Sex 
Race 
PARED 



0.01 
4.88 
2.09 



.91 

.03 
, 12 



0.06 
2:90 
4.00 



.81 

.09 
,01 



.04 

2. 18 
3.93 



.85 
.14 
.01 



N0323A 
Sex 
Race 
PARED 



1.47 
16.41 
3.49 



.23 
,00 
,03 



0.06 
■80.8.1 
3,93 



,81 
,00 
,01 



1.62 
61.15 
2.94 



.20 
.00 
.03 



NO 305 C . 
Sex 
TOC 
f^RED' 



2.21 
1.14 
12.20 



.15 
.33 
.00 



1.22 
, 6.97 
14.09 



.27 
.00 
.00 



2.67 
, 3 . 45 
16.87 



. 10 
.03 
.00 



N0317A 
Sex 
TOC 
PARED 



1 

2 
3 



0.02 
0. 19 
2.03 



,89 
,82 
13 



0.05 
1.55 
4.27 



.82 
.21 
.01 



.05 
.83 
3.99 



.82 
^44 
.01 



ERIC 



c-2 
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(continued) 



Table C-1. (continued) 

■ f 







Design Based 


Unweighted Weighted 




d.f . 


F 


Prob 


F 


Prob F ProD 






denominator d 


f.) 




(2457 denominator d.f.) 


N0323A ■ 












Sex '■<■ 


1 


.1.80 


.19 


0'.15 


.70 2.04 .15 


TOC 


2 


0.07 


.93 


2.21 


.11 0.31 .73 


PARED 


3 


3.95 


.02 


6.15 


.00 3.^1 .01 


Meaii 


(30 


denominaitor d 


• f.) 




(4895 denominator d.f.). 


Sex 


\ 1 


2.91 


.,10 


0.26 


.61 2':09 M5 


Race 


1 


'80.95 


.00 


133.84 


.00 103.89 .00 


PARED 


3 


26.08 


.00 


23.75 


.00 .27,93 .00 


Sex 


1 


2.81 


.10 


0.28 


.59 2.02 .15 


TOC 


2 


.1 . 66 


.21 


4.25 


.01 3.30 . .04 


PARED 


3 


27.98 

k3 


.00 


32.23 

0 


.00 34.12 . .00 
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Table C-2. Balanced Effect F-Tests for 13-Yeai-Olds 



T0105A 
Sex 
Race 
PARED 



d.f . 



Design Based 
F ° Prob 



(30 denominator d.f.) 



1.85 
24.27 
54.98 



.18 
.00 
.00 



Unweighted 
F Prpb 



2.91 
65.93 
29.21 



Weighted 
F " Prob 



(2416 denominator d.f.) 



.09 
.00 

.00 



2.65 
44.76 
35.58 



.10 
.00 
.00 



TOllOA 
Sex 
Race I 
PARED 



13.89 
14.49 
7.10' 



.00 
.00 
.00 



11.95 
60. 12 
9.86 



.00 
.00 

.00., 



12.66 
52.20 
9.30 



.00 
.00 
.00 



T0105A 
Sex 
TOC 
PARED 



1.68 
2.87 
56.33 



.21 
.07 
.00 



2.13 
0.98 
42.81 



.14 
.37 
.00 



2. 12 
8.23 
43.05 



.15 
.00 
.00 



TOllOA 
Sex 
TOC 
PARED 



19.01 
17.79 
11.74 



.00 
.00 
,00 



12.23 
14.46 
16.29 



.00 
.00 
.00 



13.63 
29.00 
14.09 



.00 
,00 
,00 



T0203A 
Sex 
Race 
PARED 



18.72 
27.94 
19.81 



.00 
.00 
,00 



26.60 
76.78 
25.10 



.00 
.00 
.00 



46.80 
62.74 
27.72 



.00 
.00 
.00 



T0223A 
Sex 
Race , 
PARED 



0.01 
22.27 
7.95 



.92 
.00 
.00 



0.82 
79.67 
6.9Q 



.37 
.00 
.00 



0.01 
46.15 
9.28 



.92 
.00 
..00 



TQ224A 
Sex 
Race 
PARED 



4.93 
81.61 
24.04 



.03 
.00 
.00 



7.77 
87.00 
12.42. 



.01 
,00 
,00 



9. 12 
70.14 
14. q2 



.00 
.00 
.00 



T0203A 
Sex 
TOC 
PARED 



17.37- 
0.05 
38.50 



,00 
.95 
,00 



22.75 
1.50 
41.09 



.00 
.22 
,00 



42.33 
0.3.3 
44.29' 



.00 
.72 

.00 



T0223A 
Sex 
TOC 
PARED 



0.01 
0.16 
16.66 



.93 
.85 
,00 



0.43 
2.89 
15.58 



.51 
.06 

.00 



. 0.01 
0.33 
17.32 



.94 
.72 
.00 



i3o 
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Table C-2. , (continued) 



d.f . 



Design Based 
F Prob 



Unweighted 
F Prob 



Weighted 
F Prob 



T0224A 



Sex 


1 


4.04 


.05 


6 


45 


.01 


8.02 


.00 


TOC 


2 


1.02 


.37 


5 


92 


.00 


2.10 


.12 


PARED 


3 


35.04 


.00 


25 


80 


.00 . 


27 . 30 


.00 


an 


(30 


denominator d 


f.) 






(4849 


denominator d 


f.) 


Sex 


1 


13.87 


.00 


16 


94 


.00 


22.17 


.00 


Race 


1 


59.41 


.00 


294 


95 


.00 


224.55 


.00 


PARED 


3 


57.31 


.00 


60 


97 


.00 


68.98 


.00 


Sex 


1 


14.70 


.00 


15 


.45 


.00 


21.41 


.00 


TOC 


2 


10.47 


.00 


11 


.80 


.00 


20 . 44 


.00 


PARED 


3 . 


81.71 


.00 


106 


.12 


.00 


108.98 


.00 



Table C-3. Balanced Effect F-Tests for 17-Year-Olds 

0 



d.f . 



Design Based 
F Prob 



Unweighted 
F Prob 



Weighted 
F Prob 



S01Q8A 
Sex 
Race 
PARED 



S0117A 
Sex 
Race 
PARED 



S0121A 
Sex 
Race 
PARED 



S01Q8A 
Sex 
TOC 
PARED 



S0117A 

Sex 
. TOC 
' PARED 

S0121A 
Sex 
TOC . 
PARED 

S02Q6A 
Sex 
Race 
PAREC 

S0225A 
Sex 
Race 
PARED 



(30 denominator d.f.) 



5.46 
57.77 
12.11 



0.28 
28.03 
12.30 



3.47 
1.11 
1.24 



4.99 
1.12 
34.05 



0.38 
1.89 
17.01 



3.40 
2.09 
1. 12 



17.93 
76.02 
34.91 



30.22 
121-.03 
27.81 



f.) 






(2288 


denominator 


d.f.) 


03 


6 


.07 


.01 


4.63 


,03 


.00 


111 


.52 


.00 


89.05 


.00 


.00 


9 


.93 


.00 


10.66 


.00 


Q 

60 


0 


. 10 


.75 


0.35 


.56 


00 


86 


.78 


.00 


75.18 


.00 


00 


9 


.24 


.00 


8.30 


.00 


07 


2 


.39 


. 12 


5.66 


.02 


30 


0 


.95 


.33 


1.65 


.20 


31 


1 


.87 


. 13 


1.55 


.20 


03 


5 


.25 


. 02 


3.56 


.06 


34 . 


5 


.14 


.01 


3.09 


.05 


00 


27-. 61 


.00 


25 . 79 


. 00 


54 


0 


.15 


. 70 


0 .40 


.53 


17 


1 


84 


. 16 


1.69 


.19 


00 , 


23 


20 


. 00 


18.86 


. 00 


08 


2 


25 


.13 


5.40 


.02 


14 


1 


06 


.35 


1.60 


.20 


36 ■ 


1 


89 


.13 


2.00 


.11 


00 


20. 


70 


.00 


19.28 


.t)0 


00 


111. 


94 


.00 


120.39 


.00 


00 


25. 


50 


.00 


27.88 . 


.00 


00 


18. 


22 


.00 


' 23.10 


.00 


00 


77. 


23 


.00 


86.55 


.00 


00 


17. 


50 


.00 


.. 22.25 


-GO 



SO206A 
Sex' 

' ' TOC 
PARED 



22.23 
0.22 
32.75 



.00 
.81 
.00 



27.48 
0.57- 
51.61 



.00 
.57 
.00 



23.39 
0.43 
55.71 



.00 
.65 
.00 
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Table C-3. (continued) 







Design 


Based 


Unweighted 


Weighted 




d . f . 


.F 


Prob 


F - 


Prob 


• F 


Prob 


S0225A 
















Sex 


1 


32.82 


.00 


23.59 


.00 


27.07 


.00 


TOC 


2 


,0.23 


.79 


1.39 


.25 


0.45 


.64 


PARED 


3 


37.05 


.00 


34.75 


.00 


41.03 


.00 


Mean 


(30 


denominator 


d.f.) 




(4562 


denominator 


d.fO 


Sex 


1 


23.40 


.00 


18.80 ' 


.00 . 


19.91 


.00 


Race 


1 


76.56 


.00 


270.28 


.00 


297.57 


.00 


PARED 


3 


28.58 


.00 


31.23 


.00 


37.90 


.00 


Sex 


1 


26.86 


.00 


23.43 


.00 


20.41 


.00 


TOC 


2 


5„34 


.01 


.0.71 


.49 


6.25 


.00 


° PARED 




32.90 


.00 


76.10 


.00 


84.79 


.00 . 



t 
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Appendix D 



Contingency Tables of Wald Statistic Sample Design Based Tests^ 
Versus Alternative Tests Accepted and Rejected at the 5% 

Significance Level 



.1 



134 



Table D-1. Design Based Versus SRS Linear Model Tests of NAEP Items 
for the Race -Sex - PARED Cross-Classification 



9-year-olds 



SRS 



Design Bas'^ed 

Accept Reject Total 

Accept 25 1 26 

Reject 1 5 6 

Total 26 6 32 



13-year-olds 



SRS 



Design Based 

Accept Reject Total 

Accept 20 3 23 

Reject 4 13 17 

Total 24 16 40 



17-year-olds 



SRS 



Accept 
Reject 
Total 



Design Based 



Accept 
28 
1 
29 



Reject 
2 
9 
11 



Total 
• 30 
10 
40 



All Ages 



SRS 



Design Based 

Accept Reject Total 

Accept 73 6 79 

Reject 6 27 33 

Total 79 33 112 



D-2 



o Table Design Based Versus Adjusted Linear Model Tests of NAEP Items 

for the Race - Sex * PARED Cross-Classification 



9-year-olds 

•J 

Adjusted 



13-year-olds 

Adjusted 



17-year-olds 

Adjusted 





Design 


Based 






Accept 


Reject 


Total 


Accept 


26 


1 


27 


Reject 


0 


5 


5 


Total 


26 


6 


32 




Design 


Based 






Accept 


Reject 


Total 


Accept 


21 


5 


26 


Reject 


3 


11 


^ 14 


Total 


24 


16 


40 




Design Based 






Accept 


Reject 


Total 


Accept 


28 


2 


30 


Reject 


1 


9 


10 


Total 


29. 


11 


"40 



All Ages 



Adjusted 



Accept 
Rej ect 
Total 



Design Based 



Accept 

75 
4 
79 



Reject 
8 
25 

' 33 



Total 
83 
29 
112 



ERIC 
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Table 



Design Based Versus SRS Contrast tests of NAEP Items for 
the Race - Sex - PARED Cross-Classification 



g-year-olds 



SRS 



^ Design Based 

Accept Reject Total 

Accept 18 3^ 21 

Reject 2 .13 15 

Total °20 16 36 



13-year-olds 



SRS 



Accept 
Reject 
Total 



Design Based 
Accept Reject 
13 3 



1, 
14' 



28 
31 



Total 
16 
29 
45 



17-year-olds. 



SRS 



Design Based 

Accept ' Reject Total 

Accept 15 2 17 

Reject 1 27 28 

Total 16 29 45 



All Ages 



SRS 



Design Based 

Accept Reject Total 

Accept '46 8 54 

Reject , A 68 • 72 

Total 50 76 126 



^37 













'0 




1 




P 




v 








V 






ii 


Table D- 


4. Design Based Versus Adjusted Contrast Tests 


of NAEP Items ^ 




. m 




for the Race * 


Sex * PARED Cross- 


Classif ica 


ition 




' 1 


9.-year-olds 














■ 








Design 


Based 














-» 


Accept 


Reject 


Total 






I 




Adjusted 


Accept 
Rej ect 


19 
1 


3 
13 


22 
14 












Total 


20 


16 


36 


0 




■ 


13-year-c 


>lds 




















Design 


Based 
















Accept 


Reject 


Total 










Adjusted 


Accept 
Reject 


13 
1 


4 

n 


17 
28 






1 






Total 


14 


' 31 - 


45 






1 


17-year-c 


)lds 




















Design 


leased 
















Accept 


Reject 


Total 






1 




Adjusted 


Accept 
Reject 


16 
0 


3 
26 


19 
26 












Total 


16 


29 


45 




a 




All Ages- 
















1 








Design 


Based 
















Accept 


Rej ect 


Total 










0 

Adjusted 


Accept 
Reject 


48 
2 


10 
66 


58 
68 






; 


c 




Total 


* 


76 


126 
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c 


• 
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Table D-5. Design Based Versus SRS Linear Model Tests of NAEP Items for the 
Sex * TOC - PARED Cross-Classification 



9-year-olds 



SRS 



Design Based 





Accept 


Reject 


Total 


Accept 


11 


T 


13 


Reject 


1 


2 


3 


Total 


12 


4 


16 



13-year-olds 



SRS 



Design Based 

Accept Reject Total 

Accept 16 1 23 

Reject 2 7 9 

Total .18 14 32 



17-year-olds 



SRS 



Design Based 

Accept Reject Total 

Accept 10 . 8 18 

Reject 2 12 14 

Total 12 . 20 ' 32 



All Ages 



SRS 



Accept 
Reject 
Total 



Design Based 



Accept 
37 
5 

'42 



Reject 
17 
21 
38 



Total 
54 
26 
80 



o 
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Table D-6. Design Based Versus Adjusted Linear Model Tests of NAEP 
Items for the SEX * TOC * PARED Cross-^Classif ication 



g-year-olds 



Adjusted 



13-year-olds 



17-year-olds 





Design Based 






Accept 


Reject 


1 0 ua 1 


Accept 


12 


3 


1 c 
1 J 


Reject 


0 


. 1 


1 


Total 


12 


4 


16 

- — 




Design 


Based 






Accept 


Reject 


i U La i. 


Accept 


18 


8 




Reject 


0 


6 


6 


Total 


18 


14 


32 




Design 


Based 






Accept 


Reject 


, Total 


Accept 


10 


9 ' 


19 


Reject 


2 


11 


13 


Total 


12 


20 


32 



All Ages 



Accept 
Adjusted Reject 
Total 



Design Based 



Accept 
40 , 

2 
42 



Reject 
20 
18 
38 



Total 
60 
20 
80 
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Table D-7. Design Based Versus SRS Contrast Tests of NAEP Items for the 
Sex - TOC - PARED Cross-Classification 



9-year-olds 



SRS 





Accept 


Reject 


Total 




Accept 


5 


5. 


10 




Reject 


0 


8 


8 




Total 


5 


13 


18 





13-year-olds 



SRS 



Design Based 

Accept Reject Total 

Accept 9 2 11 

Reject 2 23 25 

Total 11 25 36 



17-year-olds 



SRS 



Accept 
Reject 
Total 



Design Based 



Accept 
9 
1 
10 



Reject 
4 
22 
26 



Total 
13 
23 
36 



All Ages 



SRS 



Accept 
Reject 
Total 



Design Based 



Accept 

^ 23 
• 3 
26 



Reject 
11 
53 
64 



.Total 
34 
56 
90 



ERLC 
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Table D-8. Design Based Versus Adjusted Contrast Tests of NAEP Items 
for the Sex * TOC * PARED Cross-Classification 



Q-year-olds 



Adjusted 



13-year-olds 



Adjusted 



Design Based 





Accept 


Reject 


Total 


Accept 


5 


8 


13 


Reject 


0 


5 


5 


Total 


5 


13 


18 




Design Based 






Accept 


Reject 


Total 


Accept 


11 


7 \ 


18 


Reject 


0 


18 


18 


Total 


11 


25 


36 



17-year-olds 



Adjusted 



Design Based 

Accept Reject Total 

Accept 9 6 " ^15. 

Reject 1 20 21 

Total 10 26 ' 36 



All Ages 



Adjusted 



Design Based 

Accept Reject Total 

Accept 25 21 " 46 

Reject 1 43 44 

Total 26 64 90 
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Table D-9. Design Based Versus Adjusted Linear Model Tests for Mean Scores 
for the Race Sex - PARED Cross-Classification 



9-year'-olds 

Adj us ted 



Accept 
Reject 
Total 



Design Based 



Accept 

3 
0 
3 



Reject 
3 
2 
5 



Total 
6 
2 
8 



13-year-olds 

Adjusted 



Accept 
Reject 
Total 



Design Based 



Accept 
6 
0 
6 



Reject 
0 
2 
2 



Total 
6 
2 
8 



17-year-olds 



AdiHSted ^1^:111 
Total 



Design Based 

Accept Reject Total 

5 1 6 

0 ' 2 2 

5 3 8 



All Ages 



Adjusted 



Design Based 

Accept Reject Total 

Accept 14 4 18 

Reject 0 6 6 

Total 14 10 24 



ERLC 



D-IO 



i43 















1 


Table D- 10. ' 


e 

: Design Based Versus SRS Linear Model Tests of Mean Scores 
for the Race - Sex - PARED Cross-Classification 




1 


9-year-olds 




Design Based 






■ 


■ SRS 


Accept . 
V Reject 
Total 


Accept Reject 
3 3. 
0 .2 

r\ 5 


Total 
6 

2 . - 
-8- - ' ■• •" 






13-year-olds 




Design Based 






I 


SRS 


Accept 
Reject 
Total 


Accept Reject 
6 0 
0 2 
6 2 


Tota"l 
6 ' 
2 
8 




1 


17-year-olds 




Design Based 






I 


SRS 


Accept 
Reject 
Total 


Accept Reject 
5 1 
0 2 
5 3 


Total 
6 
2 
8 




- 


All Ages 




Design Based 


Total J 
18 
o 
24 




i 


SRS 


Accept 
Reject 
Total 


Accept Reject 
14 4 
u o 
14 10 


• 


ERLC 


0 

«■ 


o 

1 

/ 
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Table D-^11. Design Based Versus SRS Contrast Tests of Mean Scores 
for the Race " Sex " PARED Cross-Classif icatioii 



9-year-olds 



SRS 



Design Based 

Accept Reject . Total 

Accept 2 . 0 2 

Reject 0 7 7 

Total [ 1 1 9 



13-year-olds 

Design Based 



SRS 





Accept 


Reject 


Total 


Accept 


3 


0- 


3 


Reject 


0 


6 


6 


Total 


3 


6 


9 



17-year-plds 



SRS 



Accept 
Reject 
Total 



Design Based 



Accept 
2 
0 
2 



Reject 
0 

7 
7 



Total 
. 2 

7 
9 



All Ages 



SRS 



Accept 
Reject 
Total 



Design Based 



Accept 

7 
0 
7 



Reject 
0 
20 
20 



Total 
7 
20- 
27 



1 dr~ 



D-12 



Table D-12. Design Based Versus Adjusted Contrast Tests of Mean Scores 
for the Race * Sex * PARED Cross-Classif,ication 



9-year-olds 



Adjusted 



Design Based 

Accept Reject Total 

Accejit 2 1 3 

Reject 0 6 6 

Total 2 7 9 



13-year-ords . 

Adjusted 



Desigtx Based 
Accept Reject Total 



Accept 


■3 


0 - 


3 


Reject 


0 


6 


6 


'Total 


3 


6 


9 



17-year-olds 

Adjusted 



design Based 

Accept Reject Total 

Accept 2 0 2 

Reject 0 7 7 

Total 2 7 9 



All .Ages 



Adjusted 



Design Based 

Accept Reject Total 

Accept 7 1 8 

Reject 0 . 19 19 

Total 7 20 '■ 27 
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D-13 



Table D-13. Design Based Versus SRS Linear Model Tests of 'Mean Scores 
for the Sex - TOC - PARED Cross-Classif ication 



9-year-olds 



SRS 



13-year-olds 



SRS 



17-year-olds 



SRS 



All Ages 



SRS 





Design 


Based 


Accept* 

Reject 

Total 


Accept 
0 
0 
0 


Reject 
4 
4 
8 




Design Based 


Accept 
Reject 
Total 


Accept 

3 
0 
3 


Reject 
I 
4 
5 




Design 


Based 


Accept 
Reject 
Total 


Accept 
4 
1 
5 


Reject 
1 
2 

3 




Design 


Based 


Accept 
Reject 
Total 


Accept 
7 

1 . 
8 


Reject 
6 
10 
.16 



Total 
4 
4 
8 



Total 
4 
4 
8 



Total 
5 
3 
8 



Total 
13 
11 
24 



D-14 
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Table D-14. Design Based Versus Adjusted Linear Model Tests of Mean Scores 
for the Sex - TOC * -PARED Cross-Classification 



9-year-olds 



Adjusted 



Design Based 
Accept Reject Total 



Accept 


0 


7 


7 


Reject 


0 


1 


1 


Total 


0 


8 


8 



13-year-olds 



Adjusted 



Accept 
Reject 
Total 



Design Based 



Accept 
3 
0 

3 



Reject 
3 
2 
5 



Total 
6 
2 
8 



17-year-olds 

Adjusted 



Design Based 

Accept Reject Total 

Accept; 5 1 6 

Reject 0 2 2 

Total 5 - 3 ^ 



^ All Ages 



Adjusted 



Accept 
Reject 
Total 



Design Based 



Accept 
8 
0 
8 



Reject 
11 
5 ■ 
16 



Total 
19 
5 
24 



D-15 
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Table D-15. Design Based Versus SRS Contrast Tests of Mean Scores for the 
Sex - TOC - PARED Cross-Classification 



9-year-olds 



SRS 

13-year-olds 
SRS 

Q 

ly-year-olds 
SRS 

All Ages 

SRS 





Design Based 






Accept 


Reject 


Total 


Accept 


= 2 


1 


3 


Reject 


0 


6 


6 


Total 


2 


7 


9 




Design 


Based 






Accept 


Reject 


Total 


Accept 


2 


0 


2 


Reject 


1 


6 


7 


Total 


3 


6^ 


^- 9 




Design Based 






Accept 


Rej ect 


Total 


Accept 


3 


0 


. 3 


Rej ect 


0 


6 


6 


Total 


3 


6 


9 




Design Based 






Accept 


Reject 


Total 


Accept 


7 


1 


8 


Reject 


1 


18 


19 


Total 


8 


19 


27 












D-16 







Table D-16. Design Based Versus Adjusted Contrast Tests of Mean Scores 
for the Sex * TOC^ PARED Cross-Classification 



9"'year-olds 

Adjusted 



13-year-olds 

Adjusted 



17-year-olds 

Adjusted 



All Ages 

Adjusted 



Design Based 
Accept Reject Total 



Accept 


2 " 


3 


5 


Reject 


0 


4 0 


4 


Total 


2 . 


7 . 


9 


* 


Design 


Based 






Accept 


Rej,ect 


Total 


Accept 


2 


3 


5 


Reject 


1 


. 3 


4 


Total 


3 


6 


9 

* 




Design Based 


V 




Accept 


Reject 


Total 


Accept 


-3 


2 


, 5 


Reject 


0 


'4 


.4 


Total 


3 


6 


"9 




Design 


"J 

Based 






Accept 


Reject 


" Total 


Accept 


7 • 


8 


15 


Reject 


1 


11 


12 


Total 


i 


19 


■> 27 
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D-S17 



Appendix E 

Contingency Tables of Balanced Effects Sample Design 
Based Tests Versus Alternative Tests Accepted and 
Rejected at the 5% Significance Level 
r 
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•E-i 



Table E-1. Design Based Versus Alternative Tests for 



9-year-olds 



Unweighted 



Weighted 



13-year-olds 



Weighted 



17-year-olds 



Unweighted 



Accept 
Reject 
Total 





Balance Effects 






Design Based 






Accept i^eject 


Total 


Accept 


14 2 


16 


Reject 


3 11 


14 


Total 


17 13 


30 




Design Based 






Accept Reject 


Total 


Accept 


13 1 


1h 


Rej ect 


H 12 


16 


lotal 


1 T 1 O 

17 13 


30 




Design Based 






Accept Reject 


Total 


Accept 


7 - '0 


7 


Reject 


2 21 


23 


Total 


9 ' 21 


30 




Design Based 






Accept Reject 


Total 


Ac cept 


7 C\ 
1 - U 


/ 


Reject 


'2 21 


23 


Total 


9 21 


30 




V 

Design Based 


Total 




Accept Reject 



11 
1 

12 



■18 
18 



11 
19 
30 



Weighted 



Accept 
Reject 
Total" 



Design Based 
Accept Reject 



9 
3 
12 



1 
17 
18 



Total. 
10 
20 
30 



ERIC 



E-2 



Table E-1. (continued) 



All Ages 



Unweighted 



Design Based 

Accept Reject Total 

Accept 32 2 , 34 

Reject 6 ' 50 56 

Total * 38 52 90 



Weighted 



Design Based 

Accept Reject Total 

Accept 29 2 31 

Reject 9 50 59 

Total 38 52 90 



